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CHAPTER I 
INTRODUCTION AND EVALUATION DESIGN 

Introduction 

The Hartford Project Concern Program began in September of 1966 as an 
experiment in educational intervention for children from Title I schools 
concentrated in the north end of HartfordJ Receiving support from many 
areas (State of Conn<?cticut Department of Education, The Hartford Board of 
Education, The Hartford Court of Common Council, The Greater Hartford Chamber 
of Comerce, The Urban League, Community Renewal Team, The NAACP, The 
Alliance of Ministers, The PTA, The Archdiocese of Hartford, parents. Boards 
of Education from the five original participating communities, administra- 
tors, teachers, members of the legislature, and religious leaders other than 
the Alliance of Ministers or the Archdiocese of Hartford), the project 
developed se.en objectives in the original application to the Federal 
Government for funds under Title IV of the Civil Rights Act of 1964. 
These objectives were as follows: 

1. To develop a structure between a city and its suburbs 
that will desegregate schools. 

2. To discover the attitudes of children, parents, educa- 
tors, anu the conmunity when city children are bussed 
to the suburbs. 

3. To learn what happens to the educational achievement 
of both city and suburban children when city children 
go to suburban schools. 



Information relating to the history and current enrollment status of 
Project Concern was obtained from project materials. 
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4. To find out what social activities city children can 
participate in when they go to school in the suburbs. 

5. To encourage Connecticut towns to think about desegre- 
gation of schools in regional terms. 

6. To train school administrators, teachers, and aides for 
integrated schools. 

7. To find out what communities can do to make bussing 
effective. 

From 1966 to 1979, participation of suburban communities increased from 
five communities (265 children attending 35 schools) to thirteen communities 
with 1,058 students attending 75 schools. In addition, during the 1979-80 
school year 81 students attended six non-public schools in four communities 
and 289 students attended five inner-city schools in the south end of Hartford. 
Beginning with the 1980-81 school year, the Project Concern program was 
reduced. The non-public school component was eliminated and additional 
students were not allowed to enter the suburban school aspect of the program. 
In March 1983, Hartford Public School and suburban school personnel met and 
agreed to restore the enrollment in the suburban school component of Project 
Concern to 821 students for the 1983-84 and subsequent school years. 

Over the years there have been several inquiries regarding the effec- 
tiveness of Project Concern. More specifically, school boards, educators, 
and citizens in participating communities have been asking whether Project 
Concern Is successful from an educational standpoint. The difficulty In 
answering this question lies in defining the term "successful." Some accept 
the ability of students of differing races to Interact effectively as 
evidence of the success of Project Concern. Others seek measures of cogni- 
tive and affective test growth as evidence of program success. 

Two indepth inquiries into the Impact of Project Concern for the subur- 
ban, non-public and inner-city components were initiated during the 1975-1976 
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and 1976-1977 school years when the Capitol Region Education Council 
received grafts from the Connecticut State Department of Education to eval-- 
uate the program. Further information regarding the rationale and results 
of these two evaluations can be found in the documents entitled 1975-1 976 
Hailfordjroje ct Concern Evaluation Report (Iwanicki. 1976) and An Evaluation 
of_the_l97M 977 Hartford Project Concprn Pm^r.m (Iwanicki and Gable. 1977). 
Further, during the 1977-1978 and 1978-1979 project years an evaluation of 
the cognitive and affective growthof students in the suburban component was 
conducted (see An Evaluation of the 1977-197S Hartford Project Concern 
Proaian. iwa"icki and Gable. 1978, and Final Evaluation Rep ort 1Q7R-1Q7Q 
Hartford^ proi^c.t Concern Prograr.. Iwanicki and Gable, 1979). More extensive 
evaluations of Project Concern were conducted during the 1979-1980. 
1980-1961. 19eKl9£2, 1 982-1983, and 1983-1 984 school years (see Final 
Eyjlua.* Jg^ort 1979-1 980 Ha rtford Project Concern Prooram . Iwanicki and 

-S^' . ^inal Evaluation Report 1980-1 981 Hartford Project Concern 
Prosran;, iwanicki and Gable. 1981; Final Evaluation Report r -1-1982 
HarlFord^Pcoject Concern Prpgran. Iwanicki and Gable, 1982; Final Evaluatipn 
ReEPr09e^^8_3 Hartfprd Pr oject Concern Prpgram . Iwanicki and Gable. 1983; 
FinalJj^luitlon Report 1 983- 1984 Hartfprd Prpject Concern Prpgram . Iwanicki 
and Gable. 1984). Individuals interested in a summary pf the findings pf 
prior evaluations may wish tp cpnsult The Hartfprd Project Concern Prpgram ; 
A Synthesis^ t.he Evaluatipn Findings frpm 1976-1980 (Iwanicki and Gable. 
1981). 

The 1984-19^5 Project Concern Evaluatipn 

The evi»lMt1pn pf the 1984-1985 Project Cbncern ^program fpcused on the 
fpllpwing tv'o Ar^as: 
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Monitoring the cognitive and affective impact of Project Concern over 
the current school year. 

Examining the extent to which the reading and mathematics achieve- 
ment gains of various groups of Project Concern students at grades 
3, 4, and 5 were sustained over time. 

Subsequent chapters of this report provide detailed information regarding 

the evaluation design, procedures, and findings for these two areas. 
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CHAPTER II 
MONITORING THE COGNITIVE AND 
AFFiaiVE IMPACT OF PROJECT CONCERN 



Background and Evaluation Design 

For at least the last five years the funding proposal for the Project 
Concern Program has contained the following performance objectives; 

1. Pupils will show month for month gains on an average 
by grade in Language Development. 

2. Pupils will show month for month gains on an average 
by grade in M*ath. 

3. Pupils will show a positive self-concept and attitude 
toward school at the end of a year's participation. 

Up through the 1978-1979 school year, evaluations of the cognitive out- 
cor.es stated in the prcgran objectives utilized individually administered 
achievement tests (i.e., the Woodcock Reading Mastery Tests and the Key 
Math Diagnostic Arithmetic Test). These tests were administered to a random 
sample of students at grades 1-8 on a pre- to post test basis. Then, the 
res'jlts were analyzed and reported as they relate to the program objectives. 

Some disadvantages to this approach were evident. First, there were 
some problems in implementing a pre- to post test design on a yearly basis. 
By the time new participants were selected, transfers were made, project 
files were updated, and the logistics of sampling as well as pretesting were 
worked out, students were not pretested until late November or early December. 
Given that post testing must be conducted in May, there were only five to 
six months between the times of pre- and post testing. This is a relatively 
short period of time for examining pre- post test growth. 
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Secondly, although the results provided evidence of student growth, 
such grovsth could rot be compared to the growth of comparable students in 
Hartford since the same tests were not used with the general population of 
students in the Hartford Public Schools. Also, some Project Concern 
students were becoming exceedingly test wise on the Woodcock and KeyMath. 
Alternative forns of these tests we^e used on a pre- to post test basis for 
five years. Since the same level was used at grades 1-8, students at the 
upper grade levels were very familiar with the content of the text exercises. 
A final disadvantage of the approach used in past evaluations was that some 
members of the education cormiunity and the public questioned the credibility 
of results based on a random sample. 

To alleviate these problems, it was decided that the 1979-1980 and sub- 
sequent evaluations of Project Concern would monitor the cognitive perfor- 
mance of all Project Concern students at grades 2-8 cn a year-to-year basis 
using the same group adrr.i nistered achievement tests that are being used in 
the Hartford Public Schools. During the 1 981-1982 school year it was 
decided that Project Concern participants at grades 9 and 10 would also be 
tested. Appropriate levels and forms of the Metropolitan Achievement Tests 
in reading, language, and mathematics would be administered to all project 
participants in the spring according to the testing schedule used in the 
Hartford Public Schools. Results from these instruments would be analyzed 
on a pre- to post test basis (i.e., spring of one year to spring of the next 
year) and reported as they relate to the objectives of Project Concern. 

Along with the Metropolitan Achievement Tests, Project Concern students 
would also be administered a brief ten-item Student Survey. This Student 
Survey, developed for use in past evaluations of Project Concern, would be 
used to monitor Project Concern participants' attitude toward school and 
self-concept on a continuing basis. 
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Consistent with this policy for monitoring the cognitive performance cf 
Project Concern students, all participants at grades 2-10 were administered 
the appropriate level and form of the 1978 version of the Metropolitan 
Achievement Tests in the spring of 1984. At the same time, these students 
were administered the Student Survey. The Metropolitan Achievement Tests 
were administered to all students participating In the Suburban Public and 
Inner-City school components of the program. Participating suburban school 
districts accepted responsibility for testing all Project Concern students In 
ther community using the test materials provided by the Hartford Public 
Schools. 

It is important to note that during the 1979-1980 school year, the 
Metropolitan Achievement Tests were administered to suburban participants 
by Hartford Test Specialists. This approach was not used during the 
1980-1981, 1981-1982, 1982-1983, or 1983-1984 school years due to the problem 
encountered by Hartford Test Specialists. Given the time needed to administer 
the Metropolitan Achievement Tests, it was difficult to administer these 
tests to students in suburban schools without disrupting their educational 
program somewhat. In some cases students at the upper grade levels resented 
being taken away from their normal school activities to be tested, especially 
by "strangers." Students participating in the Inner-City component of the 
program were administered the Metropolitan Achievement Tests by their class- 
room teacher as part of the Hartford Public Schools spring testing program. 
Project Concern participants were tested according to the following schedule: 

Grades 5-8 March 4-15 

Grades 2-4 

and April 1-12 
9-10 
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Students were tested in the areas of reading, language, and mathematics using 
the forms and levels of the Metropolitan Achievement Tests noted below: 

Grade HATs Level 
2 Primary 2 

3-4 Elementary 
5-6 Intermediate 

2 

7-9 Advanced 1 

10 Advanced 2 

At grades 2-4 students were tested using machine scorable booklets, 
while at grades 5-10 separate machine scorable answer sheets were used. All 
tests were scored and results reported using the computer facilities of the 
Hartford Public Schools. It is important to note that only Project Concern 
Suburban participants were tested at grades 9 and 10. The Inner-City 
component of the Project Concern program does not operate beyond grades 8. In 
summary, subsequent analyses of Metropolitan Achievement Tests growth will 
focus on grades 3-10 for students in the Suburban component of Project Concern 
and on grades 3-8 for students in the Inner-City prograrr.. The number of 
Project Concern students for whoir; spring 1985 results were provided Is 
summarized b^low by grade level and prcgrair; component. 



Grade 


Suburban 


Inner-City 


2 


32 


8 


3 


49 


7 


4 


53 


20 


5 


61 


20 


6 


57 


25 


7 


90 


n 



Form 
KS 
KS 
KS 
KS 
KS 
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Grade Suburban Inner-City 

8 75 15 

9 65 
10 81 

Assessing the Achievement Growth 
of Project Concern Participants 

As noted in the prior section, the basic approach being utilized to 
assess the achievement growth of Project Concern participants is to compare 
the Metropolitan Achievi^ment Test (MAT) results from the spring of one year 
tc those for the spring of the next school year. Thus, in this year's evalua- 
tion of Project Concern, the MAT results obtained for spring 1984 and spring 
1985 were compared. In using this approach, spr-jng to spring MAT results 
must be collated by student. Some students who were tested in the spring of 
l-^es were not tested in the sprino of 1984, either because they were abser.t 
or because they were not enrolled in Project Concern at that time. The num- 
ber and percent of students tested in cne spring of 1985 for whom spring 1984 
MAT results were available is sumari-red celow by grade level and program 
compone nt . 



Grade 


Suburban 
N % 


Inner-City 
N % 


3 


28 


57 


1 


14 


4 


32 


60 


7 


35 


5 


46 


75 


17 


85 


6 


49 


86 


16 


64 


7 


66 


73 


9 


82 


8 


68 


91 


15 


100 


9 


52 


80 






10 


75 


93 
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The percent of Suburban Project Concern students tested in spring 198S for 
whom spring 1984 test scores were available increased at all grade levels In 
cdr.psrison to the 1933-1984 evaluation. This increase is an indicator of 
the increased stability of Project Concern enrollments resulting from the 
Hartford Board of Education's renewed comnitment to maintaining this program 
at an enrollment level of approximately 800 students. 

In comparing spring 1984 and spring 1985 test results, it is Important 
to note that different test schedules were used during these two administra- 
tions of the MATS. In examining the MAT achievement growth, the testing 
times and growth periods noted below should be kept in mind. 



Grade 


Spring 1984 MAT 
Testing Time 


Spring 1985 MAT 
Testing Time 


Growth 
Period 


3 


C.7 


3.7 


10 months 


4 


3.7 


4.7 


10 months 


5 


4.7 


5.6 


9 months 


6 


5.6 


6.6 


10 months 


7 


6.6 


7.6 


10 months 


8 


7.6 


8.6 


10 months 


9 


8.6 


9.7 


n months 


10 


9.7 


10.7 


10 months 



Procedures for Assessing Achievement Growth 

To assess the amount of achievement growth exhibited by Project Concern 
participants, mean standard scores were calculated by grade level In the areas 
of reading, language, and mathematics for the spring of 1984 and spring 1985 
results. Using appropriate spring nonm tables for the 1978 edition of the 
MATS, spring 1984 and spring 1985 scaled score moans were then converted Into 
mean percentile ranl:s and mean normal curve equivalent scores. The difference 
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between the spring 1984 and spring 1985 mean normal curve equivalent scores 
in the basic skill areas was used as a measure of mean growth. The results 
of these analyses are sumrr,ari2ed by grade level and program component in 
Tables 1-2. Grade 3 results were not reported for the Inner-City component 
at grade 3, since pre- post test results were available for only one student. 

In reviewing these tables, it is important to note that scaled scores 
provide a measure of student achievement in equal interval units. These 
scaled scores can be compared across forms and levels of the Metropolitan 
Achievement Tests within a particular skill ar«a. For example, the area 
of Reading for the spring 1985 testing, it is evident that sixth grade 
Suburban school students exhibited a higher level of performance (750) than 
fourth grade Suburban school students (679). It is important to note that 
scaled scores cannot be compared across skill areas. For example at grade 
4, one cannct conclude that the spring 1985 Reading performance of students 
in the Subjrban school component (679) is superior to their Mathematics 
performance (595). 

Tables 1 and 2 also contain percentile (%ile) scores. Percentile scores 
can be explained best using an example. A percentile score of 50 in 
Mathematics for grade 10 Suburban participants in spring 1985 indicates that 
on the average, their performance was better than or equal to S0% of the 
students in the norming population taking the test in the spring at grade 10. 
Percentiles are not expressed in equal interval units. The difference between 
scores at the 80th and 90th percentiles is not the same as the different* 
between scores at the 50th and 60th percentiles. Percentiles can be 
standardized (i.e., converted to equal interval units) by converting them to 
normal curve equivalents (NCE). Normal curve equivalents are also reported 
in Tables 1-2. 
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Summary by Grade Level of Mean Metropolitan Achievement Test 
SprlnR 1986 (Pre-) and Spring 1985 (^ost) Results 
for Project Concern Students 

Suburban Component 



N 


Type 

ot 
Score 




Reading 






Language 




Pre- 


Post 


Gro:*th 


Pre- 


Post 


Growth 




SS 


630 


638 


8 


525 


570 


45** 


Zo 


/•lie 




38 


-17 


55 


67 


-8 




NCE 


52.6 


63.6 


-9.0 


52.6 


68.6 


-6.2 




SS 


65Z 


679 


27** 


570 


662 


72** 




Y 1 1 A 

/•lie 


66 


62 


-6 


67 


52 


5 




NCE 


67.9 


65. S 


-2.1 


68.6 


51.1 


2.7 




SS 


676 


710 


3 ft** 


667 


678 


31** 


•»c 


/•H6 


60 


66 


ft 


56 


69 


-5 




NCE 


66.7 


67.9 


3.2 


52.1 


69.5 


-2.6 




SS 


713 


750 


37** 


687 


738 


51** 


49 


7.lle 


68 


57 


9 


52 


60 


Q 
O 




NCE 


68.9 


53.7 


4.8 


51.1 


55.3 


6.2 




SS 


729 


765 


16* 


721 


744 


23* 


66 


7. lie 


66 


66 


0 


55 


52 


-3 




NCE 


67.9 


67.9 


0 


52.6 


51.1 


-1.5 




SS 


766 


778 


32* 


753 


7 76 


23** 


68 


7. lie 


66 


51 


5 


55 


56 


-1.0 




NCE 


67.9 


50.5 


2.6 


52.6 


52.1 


-0.5 




SS 


768 


801 


33* 


788 


811 


23** 


S2 


7.1 le 


46 


56 


8 


58 


60 


2.0 




NCE 


47.9 


52.1 


6,2 


54.2 


55.3 


1. 1 




SS 


790 


798 


•3 


789 


793 


4 


60 


7.11e 


49 


42 


-7 


52 


47 


-5.0 


NCE 


49.5 


45.8 


-3.7 


51.1 


48.4 


-2.7 



Pre- 



515 
56 

52.1 

558 
66 

67.9 

617 
68 

68.9 

656 
66 

67.9 

702 
52 

51.1 

731 
48 

48.9 

773 
56 

53.2 

789 

56 

53,2 



SS-Sc«led Score; Xlle-Percentile Rank; NCE-Normal Curve Equivalent 
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Sufflinary by Grade Level of Mcnn Metropolitan Achievement T^st 
Spring I9ft4 (pre-) and Sprinj: 1985 (Post) ResuJ.ts} 
for Project Concern Students 

Tnner-City Coniponent 



Type 
of 



Reading 



Language 



N 


Score 


Pre- 


Post 


Growth 


Pre- 


Post 


Growth 


Pre- 


Post 




SS 


656 


663 


7 


S51 


682 


31 


595 


600 


7 


7.1 le 




3A 




72 


64 


-8 


63 


39 




NCE 




Al.3 


-7.6 


62.3 


57.5 


-A. 8 


57.0 


AA.l 




SS 


693 


713 


20* 


662 


661 


-1 


620 


680 


17 


7.1 le 






-I 


59 


A2 


-17 


A9 


57 




NCE 


A9.5 


US. 9 


-0.6 


5U.8 


A5.8 


-9.0 


A9.5 


53.7 




SS 


708 


7A0 


32** 


698 


730 


32- 


68A 


732 


16 


7.1 le 




52 


8 


57 


57 


0 


58 


6A 




NCE 


A6.8 


51.1 


A. 3 


53.7 


53.7 


0 


5A.2 


57.5 




SS 


717 


710 


7 


709 


705 


-A 


703 


711 


9 


Zlle 


AO 


27 


-13 


51 


38 


-13 


53 


36 




NCE 


/»A.7 


37.1 


-7.6 


50,5 


A3. 6 


-6.9 


51.6 


A2.5 




SS 


7U2 


768 


26* 


719 


7A9 


30** 


723 


765 


15 


XI le 




A6 


-2 


A3 


A3 


0 


A2 


52 




NCE 


A6.8 


A7.9 


1.1 


A6.3 


A6.3 


0 


A5.8 


51.1 



!S*Scaled Score; 7,lle"Percentlle Rank; NCE»Normal Curve Equivalent 
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An NCE of 50 is indicative of average performance for students at that 
grade level in the skill areas tested. For example, Suburban Concern pupils 
at grade 8 exhibited average performance on their post test in Mathematics 
as evidenced by an NCE of 50. To the extent that the NCE departs from SO, 
students exhibited above or below average performance in the skill area 
tested. 

Title I evaluation guidelines require that growth In the basic skill 
areas should be determined by examining the pre- and post test change in the 
mean normal curve equivalent perfonnance of the students, being served. This 
approach was utilized in assessing the achievement growth of Project Concern 
participants. In reviewing Tables 1-2, the following points should be kept 
in mind: 

a) A positive NCE gain indicates students have improved their 
relative standing regarding the national norm group. 

b) A zero NCE gain indicates the relative standing of students 
has not changed regarding the national norm group. 

c) A negative NCE gain indicates students have fallen behind 
in relative standing regarding the national nonn group. 

Findings Regarding he Cognitive Impact 
of Project Concern 

A basic question which arises in reviewing Tables 1-2 is what do these 

results tell us about the basic skill growth of Project Concern participants? 

Achievement growth can be examined on an absolute and on a relative basis. 

In assessing absolute growth, one is asking the question— how much basic skill 

growth have Project Concern students exhibited? A measure of absolute growth 

is provided by comparing spring 1984 to spring 1985 standard score means for 

each of the skill areas tested. These results are presented by grade level 

in Tables 1-2. To determine whether the spring to spring basic skill growth 

exhibited was statistically significant, correlated t-tests for the differences 
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between means were conducted. Skill areas where Project Concern participants 
exhibited statistically significant absolute growth are sunmarized in Table 
3. With the exception of grade 10, Suburban Project Concern participants at 
each grade level exhibited statistically significant achievement growth In 
most of the skill areas tested. For Inner-City Project Concern, similar 
results were evident with the exception of grades 4 and 7. 

In assessing relative growth, one is asking the question - as a result 
of the achievement progress exhibited in the areas tested, has the relative 
standing of the students changed regarding the national norm group? Percen- 
tile ranks and normal curve equivalents provide a measure of the relative 
standing of a group in relation to the national norm. As noted earlier, 
normal curve equivalents are preferable to percentiles because NCEs are 
expressed in equal interval units. The relative basic skill growth of 
Project Concern participants was determined by comparing the spring 1984 and 
spring 1985 mean NCE performance for each of the skill areas tested. These 
results are summarized in Table 4. In assessing the relative achievement 
growth of Project Concern participants based on the results presented in 
Table 4, the following conclusions can be drawn: 

Inner-City Project Concern participants tended to exhibit 
relative basic skill growth in those areas at each grade 
level where statistically significant absolute growth was 
evident. This indicates that the statistically signifi- 
cant basic skill progress exhibited by these students was 
generally reflected in an improvement in their standing 
relative to the national norm group. 

Inner-City Project Concern participants exhibited an overall 
relative NCE gain In Mathematics (+0.5) and losses In Read- 
ing (-0.7) and Language (-3.9). 

Inner-City Project Concern participants maintained or 
exhibited NCE gains in all three basic skill areas at 
grades 6 and 8, while students at grades 4 and 7 
exhibited NCE losses in these areas. Pre- post NCE 
achievement test results for students at grades 5 were 
mixed. 
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TABLE 3 

Absolute Pre- to Post Metropolitan Achievement 
Test Groh'th of Project Concern Participants 
By Skill Area! 





Suburban Component 




Grade 


Reading Language 


Mathematics 


3 






4 


♦* 




5 


♦* 




6 


*♦ ♦* 


** 


7 


* * 




8 
o 


itit 


itit 


9 


** ** 


itit 


10 








Inner-City Component 




Grade 


Reading Language 


Mathematics 


4 






5 




** 


6 


Irk 


** 


7 






8 


* ** 





^Note: * indicates absolute growth is statistically significant at the 
.05 level. 

** Indicates absolute growth Is statistically significant at the 
.01 level. 
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Table 4 



Sunmary of Mean Normal Curve Equivalent Achievement Growth by 
Grade Level, Skill Area, and Program Component for Project Concern Participants 



No. of Students Reading Language Mathematics 

Suburban Inner-City Suburban Inner-City Suburban Inner-City Suburban Inner- 

28 -9.0 -4.2 -9.6 

32 7 -2.1 -7.6 2.7 -4.8 -4.9 -12. 

^6 17 3.2 -0.5 -2.6 -9.0 -1.0 4. 

49 16 4.8 4.3 4.2 0 5.3 3. 

^ 9 0 -7 6 -7.5 -6.9 -5.9 -9. 

68 15 2.6 1.1 -0.5 0 1.1 5., 

52 4.2 1.1 4.9 

-JO . ^LZ _-3^ 

401 64 +0.6 -0.7 -0.5 -3.9 -1.2 +0.! 
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Suburban Project Concern participants tended to exhibit 
relative basic skill growth in reading at each grade 
level where statistically significant absolute growth vras 
evident. For reading, the statistically sii;nificant 
basic skill growth exhibited by students was usually 
reflected in an improvement in their standing relative to 
the national norm group. This was not the case for the 
skill areas of mathematics and language. 

Suburban Project Concern participants exhibited an overall 
relative NCE gain in Reading (+0.6) and losses in Language 
(-0.5) and Mathematics (-1.2). 

Suburban Project Concern particlpaits exhibited NCE gains 
in all three basic skill areas at grades 6 and 9, while 
students at grades 3 and 10 exhibited NCE losses in those 
three areas. Pre-post NCE achievement test results for 
students at grades 4, 5, 7, and 8 were mixed. 

To obtain further insights regarding the relative basi" skill achieve 

ment growth of Project Concern participants, NCE Reading and Mathematics 

results wer& analyzed by groupir.g students on the basis of their spring 

1984 percentile rank. Four categories we'^e formed as follows: 

23rd percentile and below 

24th - 35th percentile 

37th - 50th percentile 

51st percentile and abovs 
Mean NCE reading and mathematics growth is reported for each of these cate^ 
gories by grade level in Tables 5-6. Such data are infoniiative since they 
provide a measure of relative growth for students of different proficiency 
levels as determined by their pretest performance. From Tables 5-6 it Is 
evident that a clear relationship does not exist between students' profi- 
ciency levels and the amount of Reading and Mathematics growth exhibited. 
Some trends which emerged are the following: 

For Inner-City Project Covicem participants, students at 
or below the 36th percentile at most grade levels tended 
to exhibit the most NCE growth in Reading and Mathematics. 
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Table 5 



Summary by Grade Level and Percentile Category of 
Mean Metropolitan Achievement Test Spring 198A (Pre-) and 
Spring 1965 (Post) NCE Results for Project Concern Students 



Suburban Component 



Grade 


7.ile Category 




Pre- 


Reading 
Post 


Growth 


N 


Ma hematics 
Pre- Post 


Growt) 




23 and below 


2 


31.5 


21.8 


-9.7 


5 


25.3 


29 1 


3.8 




24-36 


2 


38.3 


38.3 


0 


2 


38.3 


39.0 


0.7 


3 


37-50 


7 


45.2 


40.7 


-4.5 


5 


45.8 


47.9 


2.1 




51 and above 


17 


59.9 


47.9 


-12.0 


16 


65.6 


45.8 


-19.8 




Total 


28 


52.6 


43.6 


-9.0 


28 


52.1 


42.5 


-9.6 




23 and belov 


1 


25.3 


33.0 


7.7 


4 


29.9 


30.7 


0.6 




24-36 


8 


40. 7 


38.3 


-2.4 


9 


40.1 


37.7 


-2.4 


4 


37-50 


9 


46.8 


44.7 


-2.1 


5 


48.9 


5-. 2 


5.2 




51 and above 


14 


55.3 


51.6 


-3.7 


13 


58.1 


46.8 


-11.3 




To ta 1 


32 


47.9 


45 .8 


-2.1 


31 


47.9 


43.0 


-4.9 




23 and below 


5 


23.0 


26.3 


3.3 


10 


30.7 


35. 3 


-7.6 




2m-36 


15 


40. 1 


41.3 


1.2 


11 


38.3 


58.3 


0 


5 


37-50 


17 


45.8 


48.9 


3.1 


7 


47.4 


42.5 


-4.9 




51 and above 


9 


59.9 


65.6 


5.7 


17 


67.7 


59.3 


-8.4 




To tal 


46 


4^ . 7 


47.9 


3.2 


45 


48.9 


47.9 


"l.O 




and Deiow 


o 
0 


25.3 


34 . 4 


9 . 1 


10 


26.3 


42.5 


16.2 






1 


39 • Q 


41.3 


2.3 


•I 

7 


38.3 


44.7 


6.4 


6 


37-50 


11 


45.8 


50.0 


4.2 


8 


4^.1 


50.5 


b.L 




51 and above 


23 


59.9 


65.6 


5.7 


24 


55.9 


6'.0 


5.1 




To ta 1 




46 . 9 


53.2 


4.3 


49 


44.7 


53.2 


8.5 




^:3 and below 


10 


25. 2 


j1 .5 


3.3 


7 


28.2 


IS. 9 


-9.3 








39.0 


36 .5 


-2.5 


9 


39.0 


36.5 


-2.5 


7 


37-50 


13 


46.3 


46.8 


0.5 


16 


44.7 


33.0 


-11.7 




51 and above 


29 


60.4 


57.5 


-2.9 


34 


62.9 


57.5 


-5.4 




Total 


66 


47.9 


47.9 


0 


66 


51.1 


45.2 


-5.9 




23 and below 


8 


26*3 


35. 8 


9.5 


8 


21.8 


30.7 


8.9 




24-36 


12 


38.3 


40 . 1 


1.8 


9 


37.7 


41.9 


4.2 


8 


37-50 


18 


45.8 


48.9 


3.1 


19 


45.8 


50.0 


4.2 




51 and aitov?^ 


27 


59.3 


58.7 


-0.6 


30 


59.3 


57.0 


-2.3 




Total 


65 


47 o9 


50.5 


2 .o 


66 


48.9 


50.0 


1.1 




23 and below 


13 


29.1 


39.0 


9.9 


7 


18.9 


41.3 


22.4 




24-36 


9 


39.0 


46.6 


7.8 


9 


38.3 


37.7 


-0.6 


9 


37-50 


10 


47.4 


48.9 


1.5 


7 


45.8 


47.9 


2.1 




51 and above 


20 


61«0 


63.5 


2.5 


29 


65.6 


69.3 


3.7 




Total 


52 


47.9 


52.1 


4.2 


52 


52.6 


56.1 


5.5 




23 and below 


17 


25.3 


30.7 


5,4 


10 


26.3 


32.3 


6.0 




24-36 


10 


39.6 


41.3 


1.7 


7 


36.5 


33.0 


-3.5 


10 


37-50 


7 


44.7 


45.8 


1.1 


X2 


46.3 


41.3 


-5.0 




51 and above 


40 


60.4 


52.1 


-S.3 


41 


64.2 


57.5 


-6.7 




Total 


74 


49.5 


45.8 • 


-3.7 


75 


53.2 


50.0 


-3.2 
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Table 6 



Suir.r.ary by Grade Level and Percentile Category of 
Mean Metropolitan Achievenent Test Spring 198^ (Pre-) and 
-^3 (Post) NCE Results for Project Concern Student! 



Spring 



Inner-City Component 



Grade 



7.ile Category 


N 


Pre- 


Reading 

Post 


Growth 


N 


Mathematics 
Pre- Post 


Grovtr. 


23 and below 


- 




- 


- 


- 


- 


- 


m 


2A-36 


I 


35.1 


37.7 


2*6 




- 






37-50 


2 


A5.8 


39.0 


-6.8 


• 

1 


45.2 


58.7 




51 and above 


4 


54.2 


43.0 


-11.2 


6 


58.7 


41.9 


_ 1 A f 


Total 


7 


48.9 


4U3 


-7.6 


7 


57.0 


44.1 


-19 0 

— 1(& . 7 


23 and below 


4 


28.2 


29.9 


1.7 


4 


29.9 


40.7 


10.8 


24-36 


3 


40.1 


44.7 


4.6 


2 


38.3 


51.6 




37-50 


4 


45.2 


46.8 


1.6 


3 


48.9 


4./ J 8 




51 and above 


6 


71.f 


62.3 


-9.5 


8 


D i . ^ 




u 


Total 


17 


49.5 


48.9 


-0.6 


1 7 


49.5 


53.7 


H . Z 


23 and below 


2 


26.3 


33.0 


6*7 


1 


34.4 


52.."^ 


27.9 


24-36 


3 


39.0 


47.9 


8.9 




40.1 


45.8 


5 7 


37-50 


5 


45.8 


46.8 


1.0 


5 


46.8 


42.5 


— H • J 


51 and above 


6 


56.7 


60.4 


1.7 


8 


64o2 


71.8 


/ • D 


Total 


16 


46.6 


51.1 


4.3 


16 


54.2 


57.5 




23 and below 


1 


26.2 


15.4 


-12.8 










24-36 


3 


36.3 


37.1 


-1.2 


1 


39.6 


32.3 


-7.3 


37-50 


4 


4A.7 


35.1 


-9.6 


3 


45.2 


33.0 


-12.2 


51 and above 


1 


81.1 


67.0 


-U.l 


5 


57.5 


50.0 


-7.5 


Total 


9 


44.7 


37.1 


-7.6 


9 


51.6 


42.5 


-9.1 


23 and below 


5 


25.3 


29.1 


3.8 


5 


23.0 


33.7 


10.7 


24-36 


3 


35.6 


25.3 


-1C.5 










37-50 


1 


46.8 


56.4 


9. •) 


5 


45.2 


47.4 


2.2 


51 and above 


6 


67.7 


68.5 


0.3 


5 


67.0 


69.3 


2.3 


Total 


15 


46.8 


47.9 


1.1 


15 


45.8 


51.1 


5.3 
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For Suburban Project Concern participants, students at 
or below the 50th percentile at most grade levels tended 
to exhibit the most NCE growth in Reading and Mathe- 
matics. 

In sufimary, both Suburban and Inner-City Project Concern participants 
tended to exhibit statistically significant basic skill growth in the areas 
of Reading, Language, and Mathematics at many grade levels. While such 
significant absolute growth was reflected In positive relative growth for 
Inner-City participants In the skill areas of Reading, Uiguage, and 
Mathematics, this was not the case for Suburban participants In Language and 
Mathematics. For Inner-City Project Concern Participants, students at cr 
below the 36th percentile tended to exhibit the most relative growth in 
Reading and Mathematics. Suburban participants at or below the 50th 
percentile tended to exhibit the most relative growth in Reading and Matne- 
matins. 

Monitoring Affective Impact 

Several research studies have been shown that affective variables relate 
to school achievement (see Bloom, Hunan Characteristics and Student Learning 
and Purkey, Self-Concept and School Achievement ). Consistent with this 
research, the Student Survey was developed during the 1977-1978 evaluation 
of Projc ,t Concern to examine the affective impact of the program. 

The Student Survey contains 10 Items which were selected from the 
Instructional Objectives Exchange nationally nomied item pool for assessing 
the areas of self-concept and attitude toward school. Given the close 
felationship between how students feel about themselves (self-concept) and 
their attitudes toward various school situations, the sat of 10 items was 
selected to generally reflect both constructs. The complete sets of self- 
concept and attitude toward school Items could not be employed as separate 
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medsures due to test length considerations. Since the items selected do 
represent the self-concept and school attitude domains, they can be employed 
validly to assess students' status. 

The Student Surve y was administered during the spring of 1985 to 
Suburban Project Concern students at grades 2-10 and to Inner-City partici- 
pants at grades 2-8 at the same time as these students were administered 

Metropolitan Achievement Tests . Table 7 contains the combined totals, 
percents and frequencies, for all Project Concern students selecting the 
"True" responses on the Student Survey . Perusal of the combined totals 
responses in Table 7 indicates that, overall, the students in Project Concerr 
continue to have positive self -concepts and attitudes toward school. This 
statement can be supported further by an analysis of the individual items if. 
the survey. Tabids 8-9 contain an iterr, by grade level stsnmary of responses 
to the Student Surv?v for Suburban and Inner-City participants. The 10 
items used i- the survey reflected three general areas: feelings about 
school and school wo.'k, attitudes toward classroom participants, and feelings 
cboL't teachers. The response? to the ite:ns tended to be consistent with the 
data from previous evaluations. 

School and School Work . The majo'-ity of students feel qui'^e comfortable 
with their school experience and their school work. Tor t^e corribined group 
of respondents, 382 indicated that they often get discourageo in school 
(item 5), and 532 lelt that they we»\? not doing as well In school as they 
would like to do (item 8). Further, B7% felt that they could get good 
grades if they wanted to (item 3), 70X felt their school work was fairly 
easy (item 1), and 81 i were proud of their schc-1 work (item 7). i'n 
addition,, only 26% of the Project Concern students felt that they were slow 
in finishing their school work (Item 6). This Is a positive finding In that 
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Table 7 

Percent and Frequency of "True" Responses 
on the Student Survey for Students Participating in 
All Corponents of the Project Concern Prograo 

N = (625)* 





I tec Stec 




Combined 


Totals 


I. 


School vork is fairly easy for me. 




70% 


(432) 


2. 


My teachers usually like me. 




897. 


(557) 


3. 


X can get good grades if I vant to. 




87v; 


(543) 


U. 


I often volunteer to do things in class. 




677. 


(418) 


5. 


I often get discouraged in school. 




387. 


(237) 


6. 


I ari slow in finishing iry school work. 




267. 


(157) 


7. 


I atr. proud of my work. 




817. 


(499) 


6. 


I ar not doing as well in school as I 
would like to. 




537. 


(329) 


9. 


I find it hard to talk in front of class. 




A 27. 


(262) 


IC. 


I don't like to be called on in class. 




257. 


(158) 



The sarrple size per item can vary slightly due to missing data. 
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Tablt 8 



Percent and Frequency of "True" Responses on the Student Survey 
By Grade Level for Students Participating in the Suburban Schools Component of 

The Project Concern Program 
(N = 540)3 



Item Stem 


2 

(N=34) 


3 

(N=49) 


4 

(N-41) 


5 

(N=62) 


Grade 
6 

(N=55) 


Level 

7 

(N=92) 


8 

(N=68) 


9 

(N=60) 


10 
(N=76) 


1 work 1^ fairly easy for 


67% 
(22) 


45% 
(22) 


55% 
(24) 


61% 
(33) 


82% 
(45) 


63% 
(58) 


74% 
(49) 


76% 
(45) • 


83% 
(63) 


achers usually like me. 


91% 
(31) 


86% 
(^2) 


89% 
(39) 


86% 
(53) 


91% 
(50) 


89% 
(80) 


94% 
(64) 


100% 
(60) 


95% 
(72) 


get good grades If I 
to. 


68% 
(23) 


57% 
(28) 


82% 
(35) 


79% 
(49) 


93% 
(51) 


95% 
(87) 


97% 
(66) 


97% 
(58) 


100% 
(76) 


en volunteer to do things in 


82% 
(28) 


78% 
(38) 


77% 
(33) 


74% 
(46) 


67% 
(37) 


68% 
(62) 


65% 
(44) 


52% 
(31) 


45% 
(34) 


en get oiscouragea in scnooi. 


Al ¥ 
*♦ 1 * 

(14) 


(25) 


Of ft> 

(16) 


(25) 


(19) 


«3H/b 

(31) 


•3 Jib 

(22) 


OCJo 

(19) 


He A» 

(31) 


slow In finishing my school 


41% 
(14) 


29% 
(14) 


33% 
(14) 


23% 
(14) 


24% 
(13) 


23% 
(21) 


21% 
(14) 


24% 
(14) 


23% 
(17) 


proud of n\y school work. 


85% 
(29) 


86% 
(42) 


86% 
(38) 


89% 
(55) 


91% 
(50) 


80% 
(73) 


78% 
(52) 


73% 
(44) 


67% 
(49) 


not doing as well In school 
Mould like to do. 


38% 
(13) 


31% 
(15) 


35% 
(15) 


57% 
(35) 


53% 
(29) 


58% 
(53) 


61% 
(41) 


70% 
(42) 


65% 
(49) 


d It hard to talk In front 
e class. 


35% 
(12) 


39% 
(19) 


52% 
(23) 


57% 
(35) 


46% 
(25) 


46% 

(42) 


27% 

(18) 


53% 
(31) 


33% 
(25) 


't like to be called on In 
• 


28% 
(9) 


27% 
(13) 


18% 
(8) 


23% 
(14) 


29% 
(16) 


25« 
(23) 


19% 
(13) 


32% 

(19) 


28% 
(21) 



B size per Item can vary slightly due to missing data. 
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Table 9 

r..w. and Frequency of "True" Responses on the Student Survey 

a. a.,, .eve, .or ""-^"-'--pat.,^^^^^^^^^^^^^ So J,X^ne„t 

(N = 85)« 



2 

(N=9) 



School work is farfly easy for me. 

My teachers usually Ifke me. 

I can get good grades ff I want to. 

I often volunteer to do things in 
class. 

I often get discouraged fn school. 

I am slow In finishing my school 
work. 

I am proud of ny school work. 



[ am not doing as well in school 
is I would like to do. 

[ find It hard to talk In front of 
:he class. 

don't like to be called on in 
lass. 



100% 
(9) 

78% 
(7) 

67% 
(6) 

100% 
(9) 

56% 
(5) 

22% 
(2) 

89% 
(8) 

44% 
(4) 

56X 
(5) 

33X 

(3) 



4 

(N=8) 


5 

(N=22) 


6 

(N=19) 


7 

(N=10) 


8 

(N=l( 


63% 
(5) 


91% 
(20) 


90% 
(17) 


eo% 

(6) 


563 
(9) 


50% 
(4) 


64% 
(14) 


95% 
(18) 


80S 
(18) 


882 
(14) 


75% 
(6) 


86% 
(19) 


100% 
(19) 


70% 
(7) 


75% 
(12) 


88% 
(7/ 


75% 
(15) 


90% 
(17) 


60% 
(6) 


63% 
(10) 


38% 
(3) 


59% 
(13) 


21% 
(4) 


30% 
(3) 


38% 
(6) 


25% 
(2) 


32% 
(7) 


21% 
(4) 


50% 
(5) 


13% 
(2) 


75% 
(6) 


86% 
(18) 


95% 
(18) 


50% 
(5) 


73% 
(11) 


50% 
(4) 


46% 
(10) 


33% 
(6) 


60% 
(6) 


44% 

(7) 


75% 
(6) 


27% 
(6) 


32% 
(6) 


50% 
(5) 


25% 
(4) 


50% 
(4) 


32% 
(7) 


11% 

(2) 


20% 

(2) 


25% 
(4) 



ample size per Item can vary slightly due to missing data. 



ERIC 



Project Concern students tend to compare themselves positively to their 
classroom counterparts in thl5= area of work completion. In comparing the 
responses of Suburban and Inner-City participants » Inner-City students 
tended to perceive their work as being easier (item 1) and were more 
satisfied with their work (item 8) than were Suburban students. 

Class Participation , The area of class participation is important as 
the Project Concern students should feel comfortable in their classroom 
setting. It appears that this is the case since 69% of the combined group 
indicated they often volunteer to do things in class (item 4). Further, 
422 felt that they found it hard to talk in front of the class (item 9) and 
only ZS% indicated that they didn't like to be called on in class (item 10). 
These figures appear typical of school children in general. 

Teachers , The students' perception that their teachers like them is 
essential for the development of healthy self-images and school attitudes. 
For the combined group of Project Concern students, 89S indicated that their 
teachers usually like them (item 2). The two groups agreed in their per- 
ception of this item. In comparing responses of Suburban and Inner-City 
participants, a higher percentage of Suburban students (91%) felt their 
teachers usually liked them than did Inner-City participants (78%). 

With respect to differences in self-concept and school attitudes across 

grade levels, some significant differences similar to previous years* data 

for Suburban participants were evident as follows: 

As grade level increased, more students tended to feel 
that school work was fairly easy for them (item 1). 

As grade level increased, more students felt they could 
get good grades if they wanted to (item 3). 

As grade level increased, fewer students indicated they 
often volunteer to do things in class (item 4). 
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As grade level increased, fewer students were _roud of 
their school work (Item 7). 

As grade level increased, more students felt they were 
not doing as well in school as they would like to do. 

For Inner-City participants, the responses differed significantly 
across grade levels for only item 1. As grade level increased, fewer 
students tended to feel that school work was fairly easy for them. 

In summary, it can be concluded that the self-concept and school 
attitudes of the Suburban and Inner-City Project Concern students in the 
areas of school and school work, classroom participation, and teachers are 
quite positive. The affective orientation of students participating in the 
1984-1985 Project Concern Program is fairly consistent with the results of 
past evaluations of Project Concern when the Student Survey was used. 
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CHAPTER III 
MONITORING THE SUSTAINED COGNITIVE 
EFFECTS OF PROJECT CONCERN PARTICIPATION 

Background 

In Chapter II we described the absolute and relative achievement grovrth 
of the Project Concern students. It Is also important to analyze whether 
the achievement gains made by Project Concern students are sustained over 
time. 

The 1983-84 Final Report presented the results of a sustained effects 
study which followed the Spring 1981 grade 3-5 Project Concern students 
through the Spring of 1984 when they were in grades 6-8. In this Chapter we 
will present the results of monitoring the trends in achievement for this 
same group through the 1984-1985 year. Complete sets of data were analyzed 
for 1981 grace 3-5 students who were in grades 7-9 at the Spring 1985 test 
time. We will also report the results of a new sustained effects study 
which will monitor the reading and math achievement levels of the 1983 grade 
3-5 students through the Spring 1985 test time. Finally, we will present 
the results of an analysis of the sustained achievement effects for a matched 
group of Project Concern and Hartford students who were in grades 3-5 during 
the 1980-1981 year and continued either in Project Concern or the Hartford 
schools through the 1984-1985 (grades 7-9) year. 
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Evaluation Questions 

The evaluation questions to be addressed In the three studies are as 
follows: 

Study I 

How are students performing who were in grades 3*5 
during the 1980-1981 year and continued in the program 
through the 1984-1985 year (grades 7-9)? 

Study II 

How are students pe?*fonning who were In grades 3-5 
during the 1982-1983 year and continued in the program 
through the 1984-1985 year (grades 5-7)? 

Study HI 

How does the achievement of Suburban Project Concern 
students, who were in grades 3-5 during the 1980-1981 
year and continued in the program through the 1984-1985 
year (grades 7-9), compare with a comparable group of 
Hartford stuoents not participating in the program? 



Regulations 

The evaluation activities are consistent with the federal law and 

regulations (ECIA, Chapter 1, Section 200.54) which state the following: 

An LEA that receives Chapter 1 funds shall, at least once 
every three years, conduct an evaluation of its Chapter 1 
project that includes. (b) a determination of whether 
improved performance is sustained over a period of more 
than one year. 

Further, the evaluation activities are consistent with the policy referenced 
in the Connecticut Chapter 1 Handbook (May, 1982, p. 42) which states the 
following: 

LEAs are required by statute to include as part of the 
evaluation plan a methodology for assessing the long 
range effects of Chapter 1 programs... 
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At least once during the three-year ap:)lication cycle the 
LEA must collect additional information needed to 
detemine whether the achievement gains measured over 6 
or 12 months are sustained over a longer period of time. 
A variety of evaluation strategies ccn be used to fulfill 
this requirement. Generally, the sustained effects study 
is based on a testing model which includes a pretest, a 
posttest, and a follow-u^ posttest. 

Ev aluation Design; Study I 

The first study actually connenced with the 1981-1982 evaluation when 

the Spring 1981 data files for grade 3-5 students were created so that the 

Spring 1982, 1983, 198< end 1985 data points could be merged into the 

overall file. These dat;a points will be used to answer the following 

research question: 

How are students performing who were in rrades 3-5 during 
the 1980-1981 year and continued in the program through 
the 1984-1985 year (giades 7-9)? 

Table 10 presents s summary of the evaluation design used to conduct 

the sustained effects study. Using the Spring 1981 Metropolitan Achievement 

Test Reading and Mathematics scores as baseline data, the design allowed the 

sustained effects of 1981 grade 3-5 students to be monitored through Spring 

1985 when these same students were in grades 7-9. Table 10 presents a 

further breakdown of the test time dimension. Displayed are the test times, 

function of the testing and files set up for the three Spring 1981 grade 

levels. Note that only students with test scores for all five test periods 

were Included In the study. The necessity of having five data points for 

each student results In sample size attrition each year the study nas been 

extended. For example. Table 11 lists the total number of students with 

test scores from the Spring 1981 files with complete data points for the 

Spring 1984 and 1985 sustained effects studies. Note that the 1985 numbers 
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Table 10 

Sustained Effects Evaluation Design: Study I 
Reading and Mathematics 



Evaluation Component 



Target Information 



A. Program Evaluation Year (baseline) 

B. Subject Areas 

C. Grade Levels (1981) 

D. Schools 

E. Test 

F. Time Period 



1981 

Reading, Mathematics 
3»4.5 

Suburban and Inner-City 
Project Concern 

Metropolitan Achievement 
Test (1978 edition) 

Spring 1981, Spring 1982, 
Spring 1983, Spring 1984, 
Spring 1985 



Time of 
Testing 



Spring 1981 
Spring 1982 
Spring 1983 
Spring 1984 
Spring 1985 



Function of 
Testing 



Data Files by Grade Levels 



Pretest 
Posttest 
Post-Posttest 
Post-Post-Posttest 
Post-Post-Post-Posttest 



File 1 
3 



File 2 
4 

5 



Pile 3 
5 

6 



8 



8 



Table 11 



Number of Complete Data Poirts for Spring 1981 
Baseline and Spring 1984 and Spring 1985 
Sustained Effects Studies by Grade Level 



Spring 

1981 
Grade 


Spring 
1981 
Baseline 


Spring 
1984 
Sustained 
Effects 


Spring 
1985 
Sustained 
Effects 


3 


72 


58 


49 


4 


84 


57 


51 


5 


89 


69 


45 



47 
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represent the number of students with five complete data points Ci»e., 
1981-1985) to be analyzed in this report. 

An Assessment of the Spring 19S2 to Spring 1985 Sustained 
Achievement Effects for Project Concern Students: Study I 

Tables 12-17 contain the MAT Reading and Mathematics data for the Total 
Project Concern group as well as the Suburban end Inner-City con?)onents. 
For each grade level the respective standard scores, percentiles, NCE scores, 
Chapter 1 NCE gains and sustained effects in NCE units are presented. 
Following each table figures are presented which contain the plots of 
respective NCE scores. 

Prior to discussing the results, a few comments regarding the interpre- 
tation of the data for a sustained effects study are in order. Each table 
lists achievement scores for five points in tir;^: Spring 1981 to Spring 
1985. Readers will note that the analysis employs NCE scores which represent 
relative growth. The scale scores listed in each table represent abso^'ute 
growth and are u^ed only to generate the corresponding percentile and Its 
associated NCE score. The first two test times (i.e.. Spring 1981 and 
Spring 1982) are used to calculate the Chapter 1 gain which is labeled 
"Gain" in each table. These gains are calculated on the basis of the 
associated NCE scores from the 1981 and 1982 data files. The focal points 
of the table are the three sustained effects (labeled "SE") from the Spring 
1982 to Spring 1983, the Spring 1983 to Spring 1984, and Spring 1984 to 
Spring 1985 test times which are based upon the difference of the respective 
NCE scores. These SE scores indicate whether the NCE gains made from Spring 
1981 to Spring 1982 are sustained over the subsequent pairs of adjacent 
years. To interpret these scores, we note that SE scores near zero would 
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Table U 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1981 to Spring 1985 
for Project Conct?rn Students 

Total Group: Reading 



Grade 
(1981) 


N 




Spring 
1981 




Spring 
1982 


Spring 

1983 


Spring Sf 
1984 ] 


3 


51 


SS 

Xile 
NCE 


644 
42 

45.8 

Gain 


1.0 


683 
44 

46.8 

SE -.± 


706 

43 

46.3 

SE 3^ 


735 
49 

49.5 , 

SE 1-1.6 1 


4 


49 


SS 

Xile 
NCE 


687 
46 

47.9 

Gain 


0 


712 
46 

47.9 

SE 4.7 


747 

55 

52.6 

SE|-3.7| 


749 
48 

48.9 

SE 1J5 


5 


42 


SS 

%lle 
NCE 


704 

42 

45.8 

Gain 


4.2 


737 

50 

50.0 

sr|-i.i| 


750 
48 

48.9 

SE -2.1 


764 
44 

46.8 

SE 4.8 
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Year Spring Spring Spring Spring Spring 

1981 1982 1983 1984 1985 

figure 1. MAT mean NCE reading scores by Spring 1981 

grade level: Total Group: Reading 
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Tahle 13 



Surmiary by Grade Level of Mean MAT Sustained 
Effects for Spring 1981 to Spring 1985 
for Project Concern Students 

Suburban Component: Reading 



Grade 
(1981) 


H 




Spring 
1981 


Spring 
1982 


Spring 
1983 


Spring 
1984 


Sp 
1 


3 


43 


SS 

Xlle 
NCE 


649 
44 

46.8 

Gain 0 


684 

44 

46.8 

SE ^ 


710 
45 

47.4 

SE 2^ 


737 

50 

50.0 

s[:[-i.i| 




4 


39 


SS 

Xlle 
NCE 


688 
46 

47.9 

Gain 0 


712 

46 

47.9 

SE 4.2 


745 
54 

52.1 

SE|-3.2| 


747 
48 

48.9 

SE JLl 


> 


5 


45 


SS 

«lle 
NCE 


704 
42 

45.8 

Gain 4.2 


737 

50 

50.0 

se|-].i 1 


750 
48 

48.9 

SE -2.1 


7S4 
44 

46.8 

SE 4.8 


7 



53 
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37 



60 



NCE 



55 



50 



45 



40 




Grade 9 

Grade 8 
Grade 7 



Grade 

Grade 3 
Grade 5 



35 



30 



Year 



i 

Spri ng 
1981 



Spri ng 
1982 



Spri ng 
1983 



Spring 
1984 



Spring 
1985 



Figure 2. MAT mean KCE reading scores by Spring 1981 
grade level: Suburban component: Reading 
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Table 14 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1981 to Spring 1985 
for Project Concern Students 

Inner-City Component: Reading 



Grade* 
(1981) 


K 




Spring 
1981 




Spring 
1982 


Spring 
1983 


Spring Spr 
1904 IS 


3 


6 


SS 

tile 
NCE 


608 
24 

35.1 

Gain 


8.5 


670 
38 

13.6 

SE 


675 
26 

36.5 

SE 8^2 


717 
40 
44.7 

SF 1-5.1 1 


4 


12 


SS 

Xlle 
NCE 


682 
44 

46.8 

Gain 


2.1 


715 
48 

48.9 

SE 5.3 


753 
58 

54.2 

SE 1-4.2 


755 
50 

50.0 

SE 2.1 



Grade 5 (1981) students deleted since there were no grade 9 (1985) Inner-City 
Project Concern students. 



56 



ERIC 



39 



60 



55 



50 



Grade B 



NCE 



45 



35 



Grade 4 



40 



Grade 3 




Grade 7 



30 



f^— 

Spri ng 
19B1 



1 — 

Spri ng 
19B2 



Year 



Spring 
19S3 



Spring 
19B4 



Spring 
1985 



Figure 3. MAT mean NCE reading scores by Spring 19B1 
grade level: Inner-City component: Reading 
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Table 15 



Summary by Grade Level of Mean MAT Sustained 
Effects for Sprinq 1981 to Spring 1985 
for Projec*: Concern Students 

Total Group: Math 



jrade Spring Spring Spring Spring 
N 1981^ 1982 1983 1984 

3 51 ss 554 613 685 694 

Xlle 44 46 47 49 

NCE 46.8 47.9 4R.4 49.5 



Gain 1.1 



Sc .5 * SE 1.1 ' $e |-7.0| 



49 SS 612 665 698 736 

$ile 45 50 51 50 

NCE 47.4 50.0 50.5 50.0 

Gain 2.6 SE .5 SCOj SE ^ 



42 SS 671 712 740 775 

«lle 53 56 53 56 

NCE 51.6 53.2 , , 51.6 53.2 

Gain 1.6 SE pTTel SE 1.6 SE 
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55 



50 



45 



N'CE 



Grade 5 




Grade 9 



Grade 8 



Grsde 4 
Grade 3 



Grade 7 



40 



35 



30 



1— 

Spri ng 
1981 



1 — 

Spri ng 
1982 



Ye<ir 



Spri ng 
1983 



Spring 
1984 



Spring 
198 5 



Figure 4. MAT mean NCE math scores by Spring 1981 
grade level: Total group: Math 
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Table 16 



Grade 

(1981) N 



ERIC 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1981 to Spring 1985 
for Project Concern Students 

Suburban Component: Math 



Spring Spring Spring Spring. Sf 

1981 1982 1983 1984 ] 



45 SS 555 610 658 693 

%lle 44 44 47 49 

NCE 46.8 46.8 48.4 49.5^^^^ 

Gain 0 SE LI SE IJ, SE |-8.2 | 

37 SS 616 657 689 735 

Xlle 47 16 47 50 

NCE 48.4 47.9 ^^-"^crn-n cv c 

Gain SE ^ SEQT] SE -._S 

• 

42 S3 671 712 740 775 

tile 53 56 53 56 

NCE 51.6 53.2 51.6 53.2 

Gain 1.6 SE R76| SE 1.6 iii 



61 62 



60 



55 



50 



45 



NCE 



40 



35 



30 



Grade 5 




Grade 9 



Grade b 



Grade 4 
Grade 3 ^ 



Grade 7 



Year 



*— — 

Spri ng 
1981 



+ 



Spri ng 
1982 



Spri ng 
1983 



Spri ng 
1984 



Spring 
1985 



Figure 5. MAT mean NCE math scorei by Spring 1981 
grade level: Suburban component: Math 



ERIC 



6\i 



Table 17 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1981 to Spring 1985 
fcr Project Concern Students 

Inner-City Component: Math 



•ade* Spring Spring Spring Spring Spring 

,981) N 1981 1982 1983 1984 1985 



3 


6 


SS 

%lle 
NCE 


541 

38 

43.6 

Gain 9^ 


635 
56 

53.2 

SE -3J_ 


662 
49 

49.5 

SE 1^ 


702 
52 
51.1 

SEl-2.2 1 


730 
48 
48. < 


4 


12 


SS 

%lle 
NCE 


600 
39 

44.1 

Gain n.8 


690 • 

61 

55.9 

SE .5 


727 
62 

56.4 

seLiiJ 


741 
54 

52.1 

SE 1.6 


777 
57 
53. i 



rade 5 (198!) students deleted since there were no grade 9 (1985) 
nner-City Project Concern students. 
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45 



60 



55 



50 



45 



NCE 



40 



35 



30 



Grade 4 
Grade 3 



Year 



1 

Spr i ng 
1981 



1 — 

Spri ng 
1982 



Grade 8 




■4- 



Spri ng 
1983 



Spr i ng 
1984 



Spring 
198 5 



Figure 6. MAT mean NCE math scores by Spring 1981 
grade level: Inner-City component: Math 
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indicate the prior jains were maintained, positive scores would indicate 
continued growth . 

Returning to Tables 12-17, we can now examine the sustained effects of 
the 1981-1982 achievement gains. For example. Table 12 presents the Reading 
data for the Total Project Concern group. The first section of the table 
lists the data for the students who were in grade 3 in the Spring of 1981. 
Fron Spring 1981 (grade 3) to Spring 1982 (grade 4) their overall Chapter 1 
gain was 1.0 NCE units. Note that since a gain of zero (0) units would 
indicate no relative growth with respect to the norm group, a gain of 1.0 
units is a positive finding. For these same students, the relative 1981-1982 
gains were not sustained over the 1982-1983 school year (SE = -.5), increased 
over the 1983-1984 year (SE = 3.2), and decreased over the 1984-1985 year, 
but did register a higher Spring 1985 relative achievement level (NCE = 47.9) 
than for the Spring 1981 data (NCE = 45.8) . At the grade 4 level the lack 
of a gain between Spring 1981 and Spring ■:982 was followed by an increase in 
the sustained effect over the 1982-1983 year (SE = 4.7). a decrease over the 
1983-1984 year (SE » -3.7), and a increase over the 1984-1985 year (SE = 1.6). 
We note also that the Spring 1985 relative achievement level (NCE = 50. g!^ is 
higher than the Spring 1981 level (NCE = 47.9) . Finally, the grade 5 
relative gain of 4.2 NCE units was followed by declines during the 1982-1983 
(SE = -1.1) and 1983-1984 (SE » -2.1) years, and an increase over the 1984- 
1985 school year (SE = 4.8). The Spring 1985 ralative achievement level is 
higher than the Spring 1981 level . 

Figure 1 presents a plot of the NCE Reading scores for the Total Project 
Concern group to illustrate the 1981-1982 gains and the 1982-1983, 1^3-1984 
sustained NCE achievement levels. Note that the horizontal line In Figure 1 
at a NCE of 50 represents relative performance which is "at grade level". 
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Comparing the Spring 1981 and Spring 1985 achievement levels indicates that 
at the Spring 1981 test time grades 3, 4 and 5 were slightly below grade 
level; by Spring 1985 grade 3 (now grade 7) was still below but closer to 
grade level and grades 4 (now grade 8) and grade 5 (now grade 9) were at or 
slightly above grade level. 

Readers are encouraged to review the Reading achievement data for the 
Suburban and Inner-City students presented in Tables 13-4 and Figures 2-3. 
Thfc Total Group, Suburban and Inner-City Mathematics achievement data are 
represented in Tables 15-17 and Figures 4-6. 

Interpretation 

The interpretation of the data presented in Tables 12-17 can be 
facilitated by focusing on two areas; the Spring 1982 to Spring 1985 overall 
sustained effects and the comparison of Spring 1981 and Spring 1385 achieve- 
ment levels to "grade l«vel" performance. 

Overall 1982-19S5 Sustained Effect Level . Table 18 presents a surma ry 
of the overall sustained effects levels for the Total Project Concern group 
as well as the Suburban and Inner-City components. To facilitate understand- 
ing of the entri-js in this table, we will focus on the overall sustained 
effect of 1.1 NCE units listed for the 1981 grade 3 students for the Total 
Group in Reading. Referring back to Table 12, we note that these grade 3 
students registered a gain of 1.0 NCE units from Spring 1981 to Spring 1982. 
The sustained effect analysis evaluated if this level of gain was maintained 
from Spring 1982 to Spring 1985. Whereas Table 12 presented the sustained 
effects of adjacent years. Table 18 presents the comparison of the Spring 
1982 and Spring 1985 achievement levels. The three year sustained effect is 



Table 18 



Overall Spring 1982 to Spring 19G5 Sustained Achievement 
Effects for Total, Suburban and Inner-City 
Project Concern Groups by Spring 1981 Grade Level 



Grade 
1981 1985 


Total ( 
Reading 


3. jup 
Math 


Suburban 
Reading Math 


Inner-City 
Reading Math 


3 7 


1.1 


-5.4 


2.1 


-5.5 


-4.0 -4.3 


4 8 


2.6 


.5 


2.1 


1.6 


3.2 2.2 


5 9 


1.6 


4.3 


1.6 


4.3 


a a 



^Grade 5 (1981) students deleted since there were no grade 9 (1985) 
Inner-City Project Concern students. 
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calculated as the sum of the three separate effects (i.e., see Table 12, 
grade 3: -.5 + 3.2 - 1.6 = 1.1) or the difference between the Spring 1985 
NCE level and the Spring 1982 NCE level (i.e., 47.9 - 46.8 = 1.1). 

By focusing on these overall sustained effects presented in Table 18. 
we can now summarize the data In Tables 12-17 on the basis of comparisons 
relative to a norm group using NCE scores as follows: 

Total Group 

Readins. Grades 3. 4 and 5 (1981) increased their 
1981-1982 gains in Reading performance. 

Math- Grade 3 (1981) exhibited an overall decline in 
Math performance as of grade 7 (1985); grade 4 (1981) 
and grade 5 (19^1) exhibited overall increases in their 
1981-'!982 Hath performance gains through grade 8 (1985) 
and grcde 9 (1985) respectively. 

MliilS.- Grades 3, 4 and 5 (1981) increased their 
1981-198t gains in Reading performance. 

Math. Grade 3 (1981) exhibited an overall decline in 
Math performance as of grade 7 (1985); grade 4 (1981) 
and grade 5 (1981) exhibited overall increases in their 
1981-1982 Math performance gains through grade 8 (1985) 
and grade 9 (1985) respectively. 

Inner-City 

Reading and Math . Grade 3 (1981) students exhibited 
an overall decline in Reading and Math performance as 
of grade 7 (1985); grade 4 (1981) students exhibited 
overall increases in their 1981-1982 Reading and Math 
performance gains as of grade 8 (1985). 

In summary, these findings suggest that for the Total Project Concern 

group 1981-1982 Reading achievement gains were sustained and increased for 

the 1982-1985 period for the three grade levels studied. This was also the 

case in Math for the 1981 grade 4 and 5 students, but not for the 1981 grade 

3 students. Since we know that sustained effects at the 0.0 (I.e., no 

change) level indicate that achievement h6s been maintained at a level 
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relative to growth of the norm group, the NCE growth Indices in Table 18 are 
positive indications of student growth above the performance of the national 
norm group. The overall declines in the area of Math for 1981 grade 3 
students in the Suburban and Inner-City groups should be discussed by project 
staff. 

To facilitate this discussion, we note that the decline in Math 
performance was most recently evident for the grade 3 (1981) Suburban students 
(see Table 16) during grade 7 (i.e.. Spring 1984 to Spring 1985); the most 
recent declines in both Reading and Mat.i performance for ths y^-'^^ 3 (1981) 
Inner-City students were also during grade 7 (see Tables U 17). jhis 
trend for a decline in performance during the grade 7 middie school transi- 
tion may be expected by some and posibly easily explained. We merely point 
out that Table 18 depicts the possible trend for the overall three year 
1982-1984 sustained effects data. 

Further inspection of the adjacent year sustained effects indices in 
Tables 12-17 suggests that declines in both Reading and Math performance 
were consist^^ntly present for the Spring grade 7 data (see the SE indices in 
boxes). These grade 7 declines were noted in comparison to the prior Snring 
grade 6 data which consistently depicted increases in relative achievement 
performance levels during grade 6 (see Tables 12-17). The only exception 
was found in Table 16 for the grade 4 ("981) Spring 1983 to Spring 1984 data. 

To illustrate this point, we will review the data in Table 12. The 
three grade 7 sustained effect indices relevant for 1981 grades 3, 4 and 5 
are located respectively under Spring 1984 - Spring 1985 (grade 3, -1.6), 
Spring 1983 - Spring 1984 (grade 4, -3.7), and Spring 1982 - Spring 1983 
(grade 5, -1.1). Readers will note that the three respective sustained 
effects NCE indices all suggest a decline during grade 7 following an 
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increase during grade 6. Readers can locate these same trenas in Tables 
13-17 (see SE indices in boxes). These findings could have implications for 
curricula, instruction and testing areas. Ordinarily we would suggest that 
Hartford staff may wish to discuss the match among the stated curriculum: 
what is taught, and how well the MAT assesses what is taught at the grade 7 
level (i.e.. the issues of instructional and curricular validity). In this 
situation several different school systems are involved so it is difficult 
to analyze the curricular and testing implications of these findings. It 
may be that the nature of the MAT norm group contributes to this trend in the 
data. Some educators have stated that the Hartford students mature at an 
earlier age than the general population represented in the norm group. This 
earlier maturity may often result in emphasis on social relationships during 
the grade 7 middle school experiences. Comparisons of the Hartford students 
with a national norm group could result in an apparent decline in grade 7 
and possibly rade 8. relative achievement levels. During grada 9 ths 
comparison with the national norm group would most likely res.it in positive 
sustained achievement effects indices. Readers can examine the Spring 1331 
grade 5 data in Tables 12, 13, 15 and 16 where this was indeed the c.se for 
the Spring 1985 grade 9 students. The Spring 1984 to Spring 1985 sustained 
effect (SE) indices in the lower right side of t' -se tables (underlined with 
a bold line) are all positive values indicating ar>-^3vement growth above the 
level exhibited by the overall norm group. 

Grade level performance. , Grade level performance comparisons also 
assist in interpreting the sustained effects data presented in Tables 12-17. 
Recalling that performance at the 50th Xile or above indicates "grade level 
or above" relative achievement perfonnance, the Total Group, Suburban, and 
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Inner-City Spring 1981 and Spring 1985 data were examined. Table 19 presents 
a sumnary of the findings. A plus sign (+) indicates that the relative 
achievement level was at or above grade level at the time of testing; a 
minus (-) sign signifies below grade level perfonnance. 

Inspection of the data for the Total Group, Suburban and Inner-City 
components indicates that, for these cohort groups studied over the four 
year period, achievement has been improving on an overall basis. For the 
Total Project Concern group comparisons for three grade levels and two 
content areas (i.e., six comparisons) suggest that three of the comparisons 
(i.e., Reading, 19S1 grades 4 and -5; Math, 1981 grade 4) were associated 
with below grade level Spring 1981 performance and above grade level Spring 
1985 performance. For no comparisons were the Total, Suburban or Inner-City 
groups associated with a trera fror;: above (Spring 1981) to below (Spring 
1985) grade level performance. These overall trends are depicted in 
Figurts 1-6 where points above a NCE of 50 (i.e., the 50th %ile) indicate 
above grade level performance. 

Summary 

This section has presented the results of a "-ustained achievement effects 
study for Spring 1981 grades 3-5 students who were in Project Concsm as of 
Spring 1985 (grades 7-9). Overall, the findings support the claim that 
Project Concern students have generally maintained and enhanced their achieve- 
ment performance over time. The exception to this general trend was found 
for all three cohort groups during grade 7. 
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Table 19 



Grade Level Achievement Perfonnance Status 
for Spring 1981 and Spring 1985 Cohorts 
by Project Concern Group and Grade 



MAT 1981 Achievement Relative to Grade Level* 

Group Area Grade Spring 1981 Spring 1985 

Total Reading 3 - 

4 - ♦ 

5 - + 

Math 3 - - 

4 - ♦ 

5 + + 



Suburban Reading 3 . 

4 - + 

5 - + 



Math 



4 
5 



Inner-City 



Reading 
Math 



3 
4 

3 
4 



*Grade level performance represents a NCE of 50 or the 50th percentile. 

{-) = Below grad'ii level 

(+) = At or above grade level 

^Grade 5 (1981) data not presented since there were no grade ^' (1985) 
Inner-City Project Concern students. 
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Ivaluation De s ign: Study 11 

The second sustained achievement effects st^d/ addressed the following 

research question: 

How are students performing who were 9rade$ 3-5 
during the 1982-1983 year and continue^ the 
program through the 1984-1985 year (gr^d^s 5-7)? 

Table 20 presents a summary of the evaluation Resign used to conduct the 
second sustained effects study In the areas of P^a^ing and i^athanatlcs. 
li^in^ the Spring Metropolitan Achievement Test Reading and Mathematics scores 
as baseline data, the design allowed the Spring IgS^ * Spring 1984 gains to 
be examined for sustained achievement effects frOm Spring 1984 to Spring 
1985, Table 20 also presents a breakdown of the t^st tine dimension. Dis- 
played are the test times, function of the testing ^r\d files set up for the 
three grade levels. Note that only students wit^ ^feSt scores for all three 
test periods were included in the study. 

An Assessment of the Spring 1984 to Spring 198S^ S!i$ tained 
jjchlevement Effects for Project Concem Students r"l|fu^^ 

Tables 21-26 present the MAT Reading and Mrjthetjiatics data for the Total 
project Concem group as well as students in the SUbiirban and Inner-City 
coinpanents. Figures 7-12 present plots of the achl^/einent data. Readers 
i^e referred to the section of Study I presented e^H ier in this chapter for 
» description of the table entries and the Interpretation strategy for gains 
md sustained effect indices. 

On the basis of the data presented in Tables and Figures 7-12 

the following sunriary of achievement levels relative to a nom group is 
presented: 
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Table 20 



Sustained Effects Evaluation Design: Study II 
Reading and Mathematics 



Evaluation Component 



Target Information 



A. Program Evaluation Year (baseline) 

B. Subject Areas 

C. Grade Levels (1983) 

D. Schools 

E. Test 



1983 

Reading, Mathematics 
3.4,5 

Suburban and Inner-City 
Project Concern 

Metropolitan Achievement 
Test (1978 edition) 



F. Time Period 




Spring 1983, Spring 1984, 
Spring 1985 


Time of 
Testing 


Function of 
Testing 


Data Files by Grade Levels 






File 1 File 2 File 3 


Spring 1983 


Pretest 


3 4 5 


Spring 1984 


Pcsttest 


4 5 6 


Spring 1985 


Post-Posttest 


5 6 7 
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Table 21 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1983, Spring 1984 and Spring 1985 
for Project Concern Students 









Total Groups: 


Reading 




Grade 
( 1983) 


N 




Spring 
1983 


Spring 
1984 


Spring 
1985 


3 


36 


SS 

%ile 
NCE 


643 
41 

45.2 

Gain 


675 
40 

44.7 

-0.5 SE -0.6 


699 
39 

44.1 


4 


48 


SS 

%lle 
NCE 


697 
51 

50.5 

Gain 


720 
52 
51.1 

0.6 SE 3.1 


754 
58 

54.2 


5 


62 


SS 

%lle 
NCE. 


704 
42 

45.8 

Gain 


733 
48 
48.9 

3.1 SE -2.6 


740 
43 
46.3 



7; 



57 



NCE 



60 



55 



45 



40 



35 



30 



Grade 5 
Grade 3 g 



Grade 6 




4- 



Spr i ng 
1983 



■4- 



Spr i ng 
1984 



— I — 

Spn ng 

1985 



Figure 7. 



MAT mean NCE reading scores by Spring 1983 
grade level: Total group : Reading 
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Table 22 



Summary by Grade Level of M^an J Sustained 
Effects for Spring ?5?:3, Sprvot ^v^' and Spring 1985 

for Project Concos'^ ^ : <dents 

Suburban Component: Reading 



Grade 
(1983) 


N 




Spring 
1983 




Spring 
1984 


Spring 
1985 


3 


28 


SS 

%lle 
NCE 


636 
38 
43.6 

Gain 


-2.3 


664 
34 

41.3 


694 

36 

42.5 


4 


32 


SS 

%ile 
NCE 


705 
55 

52.6 

Gain 


-0.5 


726 
54 

52.1 

SE 4^ 


761 
62 , 
56.4 


5 


93 


SS 

%lle 
NCE 


708 
44 

46.8 

Gain 


3.2 


736 
50 

50.0 

SE -2.6 


744 
45 
47.4 
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• Grade 6 



Grade 4 



Grade S 



Grade 7 



Grade 3 



Grade 5 



1— 

Spring 

1983 



■ K- 

Spri ng 

1984 



4. 



Spri ng 
1985 



Figure 8. MAT mean NCE reading scores by Spring 1983 
grade level: Suour&an component : Reading 
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Table 23 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1983, Spring 1984 and Spring 1985 
for Project Concern Students 

Inner-City Component: Reading 
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MQftfi N ^P'^'"9 Spring Spring 

N 1983 1984 1985 



8 SS 669 713 

59 

I 54 
Gain 1^ SE -5.3 

16 SS 680 708 

44 

) 46 
Gain 0.5 SE 4.3 



« 716 

xiie 56 59 49 

NCE 53.2 54.8 49 5 



tile 43 44 52 

NCE 46.3 46.8 51.1 



SS 681 717 710 

%lle 30 40 27 

NCE 39.0 44.7 37.1 
Gain 5.7 SE -7.6 
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I — * u- 

Spring Spring Spring 

1983 1984 1985 

Figure 9. HAT mean NCE reading scores by Spring 1983 
grade level: Innsr-ctty component: Reading 
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Table 24 



Suiimary by Grade Level of Mean HAT Sustained 
Effects for Spring 1983, Spring 1984 and Sprang 1985 
for Project Concern Students 

Total Group: Math 



Grade Spring Spring Spring 

(1933) N 1983 1984 1985 

3 36 SS 541 608 649 

%lle 37 43 42 

NCE 43.0 46.3 45.8 

Gain 3.3 SE -0,5 



48 SS 620 676 725 

%lle 49 55 62 

NCE 49.5 52.6 56.4 

Gain 3.1 SE 3.8 



62 SS 658 70 1 712 

%lle 47 52 36 

NCE 48.4 51.1 42.5 

Gain 2.7 SE -8.6 
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63 



60 



55 " 



sn 



45 



Grade 6 



Grade 5 



Grade 



NCE 




-Nr 0 Grade 5 



Grade 7 



40 



35 



30 



+ 



Spri ng 
1983 



+ 



Spri ng 
1984 



•4^ 



Spri ng 
1985 



Figure 10. 



MAT mean NCE math scores by Spring 1983 
grade level; Totsl group: Hath 
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Table 25 

Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1983, Spring 1984 and Spring 1985 
for Project Concern Students 

Suburban component: Math 



Sprang Spring Spring 

1983 1934 1985 



Grade 

( 1983) N 



28 



SS 

%lle 
NCE 



527 

32 

40.1 



Gain 4.0 



599 
39 

44.1 



SE -M 



638 
37 

43.0 



4 



32 



SS 

t'.ie 
NCE 



615 
.^7.9 



Gain 3.7 



671 
53 

51.6 



SE 3.7 



721 
60 

55.3 



53 



SS 

%lle 
NCE 



• • p. 

'if 

48.4 



Gain 2.7 



701 
52 
51.1 



SE -d.6 



712 
36 

42.5 



«5 
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T 



' 1 1 1_ 

Spring Spring Spring 

1983 1984 1985 

Figure 11. MAT mean NCE math scores by Spring 1983 
grade level: Suburban component: Math 
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Table 26 



Summary by Grade Level of Mean MAT Sustained 
Effects for Spring 1983, Spring 1984 and Spring 1985 
for Project Concern Students 

Inner-City Component: Math 



« ^P'''"^ ^P'"'"9 Spring 

(1983) N 1983 1984 1985 



8 SS 588 637 691 

%lle 60 58 61 

NCE 55.3 . 54.2 55.9 

Gain -1.1 SE 1.7 



16 SS 630 684 732 

"^lle 54 58 64 

NCE 52.1 54.2 57.5 

Gain 2.1 SE 3.3 



SS 657 703 711 

«ile 46 53 36 

NCE 47.9 51.6 42.5 
Gain 3.7 SE -9.1 
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Spring Spring Spring 

1983 1984 1985 



Figure 12. MAT mean HCE math scores by Spring 1983 
grade level: Inner-City component; Math 



ERIC 



68 



Total Group 

Reading and Math . Grade 4 students increased their 
|«SJ-!»B4 gain during the 1 984-": 985 year; grades 3 
and 5 students exhibited decreases in their 1984- 
1985 achievement levels when compared to their 
1983-1984 level. 

Suburban 

Reading . Grades 3 and 4 students increased their 1984- 
1985 achievement levels after exhibiting a decrease 
during the 1983-1984 period. Grade 5 students failed 
to maintain their 1983-1984 gains during the 1984-1985 
year. 

Math. Grade 4 students Increased their 1983-1984 gains 
during the 1984-1985 year. Grades 3 and 5 students 
failed to maintain their 1983-1984 gains during the 
1934-1985 year. 

Inner-City 

Reading . Grade 4 students increased their 1983-1984 
gains during the 1984-1985 year; grades 3 and 5 
students failed to maintain their 1983-1984 gains 
during the 1984-1985 year. 

Math. Grades 3 and 4 students exhibited gai:.- during 
T5S^T-1985 as compared to their 1983-1984 achievement 
levels; grade 5 students failed to maintain their 
1983-1984 gains during the i:.84-1985 year. 



I nterpretation 

The sustained effects data presented in the previous section represents 
a one year (Spring 1984 to Spring 1985) period. When compared to the four 
year period of data presented In Study I, these sustained effects results 
are considered to be for a relatively short period of time, it was found 
that over the one year (Spring 1984 to Spring 1985) period several of the 
Spring 1983 to Spring 1984 gains were not maintained. It will be necessary 
to follow these same students over a longer time period before any meaning- 
ful trends In achievement can be noted with confluence. 



er|c ^0 



Sustained Effect Level . On an exploratory basis, though, we do note 
that for the Total Project Concern and Suburban groups the Spring 1985 
achievement levels were higher than the Spring 1983 levels for four of six 
comparisons (Reading, grades 4-5; Math, grades 3-4). For the Inra; -City 
students Spring 1985 achievej-ent levels were higher than Spring 1983 levels 
for three of six comparisons (Reading, grade 4; Math, grades 3-1). 

Grade Level P erformance . Grade level performance status (i.e., above 
or below the 50th ^ile) for the Spring 1983 data and f*- "oring :935 data 
teas ?ilso examined (see Tables 21-26). For most compariscj the grade 3, 4 
and 5 (1983) students maintained their achievement level status with respect 
to grade level performance. For example, the Total Project Concern grade 3 
students were below grade level or the 50th %ile (41st !6ile) for Spring 1983 
Reading and still below the 50th %ile (39th ^ile) in grade 5 at the Spring 
1985 test time (see Tabla 21). The positive exception to this situation was 
found for the Total Group, Suburban, and Inner-City students for the Sp'-ing 
1983 grade 4 Math data, where the Spring 1983 below grade level performance 
was found to be above grade level by Spring 1985 in grade 6. 

Summary 

This section pre^tuzad the results of a sustairjed effects study for 
Spring 1983 grades 3-5 students who were in Project Concern as of Spring 
1985 (grades 5-7). Noting that the one year period used to initiate the 
study represented the beginning of an ongoing sustained effects evaluation 
design, it was found that in several cases the Spring 1983 to Spring 1984 
gains were not maintained, it was found, though, that in sr^veral cases the 
Spring 1985 achievement levels were higher than the Sprinc 1983 levels, but 
that stut^erits tended to maintain thiir grade level perfonnance status. 
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This cohort Qroup of 1933 grades 3-5 students will have to be followed for 
a longer period of time so that valid trends can be identified in i-ngttu- 
dinal achievement performance levels. 

An Assessment of the Sprin'j 1982 to Spring 1985 Sustained 
Achievement Effects for Suburban Project Concern~and' 
Hartford Non-Participants; Study III 

At the request of the Hartford Board of Education, a sustained achieve- 
ment effects study was carried out to address the following questior.: 

How does the achievement of Suburban Project Concern 
students, who were in grades 3-5 during the 1980-1981 
year and continued in the Program through the 1984- 
1985 year (grades 7-9),' cc ,,are with a comparable 
group of Hartford studenti- not participating in the 
program? 

E valuation Design 

The evaluation design employed to answer this question is similar to the 
traditional sustained achievement effects desigr. employed in Studies I and 
11 reported in the previous section. The unique future of Study III will be 
the emphasis on the statistical significance of the differences in the 
achievement levels of Project Concern and Hartford comparison students at 
each of the 1981 to 1985 test times. 

Table 71 contains the sustained effects design which specifies that the 
MAT Reading and Math levels of the Suburban Project Concern and Hartford 
Non-Participants would be followed for 1980-1981 grades 3-5 students in the 
program through the 1984-1985 (grades 7-9) year. Inckided in the table aro 
the testing times, function of tha tests and the data files created by 
grade level. 
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Table 27 

Sustained Effects Evaluation Design: Suburban vs. 
Hartford Non-Participants Reading and Mathematics 



tv'sluation Component 



Target Infomatton 



A. Program EvaluatJcn Year (bcselir 2i 

S. Subject Areas 

C. Grade Levels (1981) 

D. Schools 



E. Test 



F. Time Period 



1981 

Reading, Mathematics 
3.4.5 

Suburodn Project Concern, 
Hartford No n- Participating 
Students 

Metropolitan Achievement 
Test (197S edition) 

Spring 1981, Spring 1982, 
Spring 1983, Spring 1984, 
Spring 1985 



Time of 
Testing 



Spring 1981 

Spring 1982 

Spring 1983 

Spring 1984 

Spring 1985 



Function of 
Testing 



Data Files by Grade Levels 



Pret -"t 
Post' ; 
Pcst-P St -est 
Post-Post-Posttest 
Post-Post-Post-Posttest 



File 1 
3 

4 



File 2 
4 

5 



File 3 
5 

6 



8 



8 
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Selecting the Hartford Comparison Group 

The validity of the sustained effects study rests heavily on the proper 
selection of a "matched" group t* Hartford students who were similar to the 
Suburban Project Concern students on relevant variables, but did not 
partif^ipate in the program. The methodology for creating the comparison 
^^^^^ was as follows. Rosters were created which contained the names and 
19eO-1981 MAT Reading and Math scores for all Suburban Project Concern 
students who were in grades 3-5 during the 1980-1981 school year. The 
Project Concern administrators then filled in each student's ethnic group, 
sex, year and grade they entered the program, and the name of the Hartford 
sending school J The Hartford Public Schools Evaluation, Research and 
Testing office screened the 1980-1981 achievement rosters for each of the 
sending schools to locate names of students with the same sex, et^;nic group, 
and achievement scores. To facilitate the achievement score matches, error 
bands of +2 standard errors of measurement were placed around e^ch Project 
Concern student's Reading and Math scores. T::e error bands were to create 
95% confidence intervals around the Project Concern students* scores so that 
the Hartford Non-Participa' - comparison grj'i^i^ w sjld not d. far significantly 
from the Project Concern students in their 15? : - ; ac -^vi^fnent levels. 

In searching the sending school 1980-1981 files for rriatches, it became 
apparent that there would be no matches for some of the Project Concern 
students. Therefore, a third group was created for ^^ysis in the sustained 
effects study: the Project Concern Non-Match group. TabT.- 28 lists the 
number of Project Concern students in grades 3-5 during the 1980-1981 year 



The following Hartford sending school i: were involved: Barnard Brown, 
Barbour, Clark, Fisher, Hooker, Jones, King, Mark Twain, Rawson, Waverly, 
Wish and Vine. 
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Table 28 

Number of Students In the Matched Pre 
Hartford Non-Partic jpant, and Project Conc<? 



^oncern, 

tch Croups 



1980-1981 
Grade 


Nuaber {n 
Project Concern 


humber of 

Project 

Concern 


hatched Students 
Hartford 
Non-Part1cin«nt5 


yer of Non 
P. ^ect Concern 


-Matr ed 

Stu ./ 


Percent of 
Hatches 


3 


43 


26 


26 


17 




60. 5X 


4 


39 


18 


18 


21 




46. 2X 


5 


45 


27 


27 


18 




60. OX 


Total 


127 


71 


71 


56 




56. 9X 



Represents the nunber of students with 
coaplete data files froa 1980-1981 to 
1984-1985, 
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with complete sets of achievement scores through the current 1984-1985 year. 
Of these 127 students, 71 or 55.9% were found to have matches in the Hartford 
s^n^ing schools based upon the achievement level, sex and ethnic group 
fi^fte^^a. A iotal of 56 Project Concern students could not be matched and 
t<^r^ includei^ 1n the Project Concern Non-Match group. Table 29 presents a 
t)^ca|c<lown of the ethnic group and sex of the three groups studied. 

The attv'Ifcutc of the Project Concern Non-Match group which precluded 
notching themJ with Hartford Non-Participants was a tendency for higher 
pftading or M^th achievement levels than the Hartford Non-Participants and 
the Project C(>ncern "matched" students. While this apparent trend for higher 
achie>/enient l^wels precluded matching several of the individual Project 
concern and Hertford students, we will present data later which docL-ments 
th^t the differences were nre^y significant from a statistical perspective. 
F^o^ Xf]^ 1980'19»81 data the Project Concern Non-Match group had significantly 
(P-^.OS) higher achievement levels than the Hartford comparison group for 
9r^d^ 3 in Maifr: no differences existed for grade 3 in Reading or grades 
4 ind $ in Resiling and Hath. Further, the Project Concern Non-Match had 
h-fghCr achievement than the Suburban Project Concern "'matched" grcup for 
9f4, ; in Reading J re differences were found for grade 4 in Math or grades 
3 5j 5 in Reading and Math. The essential point here is that the differ- 
ences in achlf^vement levels frtcluded matching some individual students, but 
tfjft differences between the respective groups were not often significant on 
« Statistical basis. 

in explatuflng the reason for existence of the Project Concern Non-Match 
SKOup, we not^f that the 1980-1981 grade 3 students entered the program from 
1377 \o 1980» the grade 4 students from 1976 to 1980, and the grade 5 
students from 1975 to '980. These entering years are relevant in light of 
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Ethnic Groups and Sex cf Sto»1ents 



Matched Students 
Grade Project Concern Non-Matched Participant s 

Frequency Frequency 



Non-Hatched 
Project Concern Students 
Frequency 



Total 
Frequency Pi 



f\\c Group 
Caucasian 



Hispanic 



2 
I 
7 



2 
1 

7 

10 



Black 



3 
4 

5 



26 
18 
27 



Z6 
18 
27 



14 

20 
11 



66 
56 

65 
187 



-enale 



3 
4 

5 



17 
15 
16 



5 7 
15 
16 



9 
9 
12 



43 

39 
44 

126 



ale 



3 
4 

5 



9 
3 
II 



9 
3 
11 



8 
12 
6 



26 
18 
28 

72 



96 



97 
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the analysis of Project Concern selection procedures from 1966 to 1980 
which were documented in the 1980-1981 evaluation report entitled 
Final Evaluation Report: 1980-1981 Hartford Project Concern Program 
(Iwanicki and Gable, 1981). In this report we documented how the random 
selection procedures originally employed in 1966 changed during the late 
1970's to become essentially a volunteer procedure by 1980. It is thus not 
surprising that a group of Project Concern students existed which could not 
be matched with the Hartford Non-Participants. 

At this point the validity issue must be addressed. Is it feasible to 
monitor the Matched Project Concern and Hartford Non-Participant groups if 
the Project Concern group includes only 56.9% of the 1980-1981 grade 3-5 
students? The answer to this question is clearly "yes", as long as we 
separate out the Project Concern Non-Matched students. In this manner we 
have actually controlled for the differential selection procedures which may 
have occurred, and will be studying more comparable Project Concern and 
Hartford Non-Participants. As a point of interest, we will maintain the 
Project Concern Non-Match group and will follow their achievement across the 
5-year period. 

Data Analysis 

The comparison of the Project Concern and Hartford matched student data 
consisted of running t-tests for each of the 1981 to 1985 years. To examine 
the differences among the Project Concern, Hartford, and Project Concern 
Non-Match groups one-way analyses of variance followed, where necessary, by 
Scheffe' tests were carried out. 
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Findings 

This section will present the findings regarding the comparison of the 
achievement levels of the Suburban Project Concern, Hartford, and Project 
Concern Non-Hatch groups. After presenting some overall findings, the 
results will be described separately for the grade 3, 4 and 5 1980-1981 
students. The primary focus will be placed upon comparing the Suburban 
Project Concern and Hartford students. Tables 30-32 present the Spring 1981 
to Spring 1985 MAT Reading and Math data for the three groups examined. 
Included are the MAT standard scores (SS), percentiles (%ile) and normal 
curve equivalent (NCE) scores. Figures 13-18 present plots of the NCE Read- 
ing and Math scores for the 1981-1985 period beginning with the 1980-1981 
grade 3, 4 and 5 groups. 

Suburban Project Concern and Hartford Comparison Group Spring 1981 
Baseline Achievement . The analysis of 1980-1981 "baseline" (i.e., starting 
point) data indicated that there were no significant differences between the 
Project Concern and Hartford Matched students in Reading or Math. These 
results simply confirm that the selection of the Hartford comparison group 
on the basis 1980-1981 achievement levels was properly conducted. The 
achievement level of both groups was near the 40th percentile, which is 
considered to be below grade level (a NCE of SO or the 50th %ile indicates 
perforrance at grade level). 

Suburban Project Concern and Hartford Comparison Group Spring 1982 to 
Spring 1985 Achievement . After confirming that the Suburban Project Concern 
and Hartford comparison students did not differ in their Spring 1981 Reading 
and Math levels, the analysis could turn to the Spring 1982 to Spring 1985 
longitudinal data. As presented in Tables 30-32 and depicted in Figures 
13-18, the major finding of this study was as follows: 
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Table 30 

Spring 1981 to Spring 1985 MAT Scores for Project Concern 
and Hartford Matched Students and Project Concern Non-Matched Students 

Grade 3 (1980-1981 ) 



Group 



Spr i ng 
1981 



Spri ng 
1982 



Spr i ng 
1983 



Spri ng 
1984 





SS 


636 


6?2 


695 


731 


Proipct CnncArn 


Xile 


38 


38 


27 


47 


(N=26) 


NCE 


44 


44 


43 


48 






U J u 


U V H 


689 


724 


nd r t T u 1 u 




O o 






43 


(N=26) 


NCE 


44 


42 


41 


46 




SS 


659 


695 


725 


744 


PC Non-Match 


%lle 


50 


50 


54 


53 


(N=17) 


NCE 


50 


50 


52 


52 




SS 


534 


594 


642 


676 


Project Concern 


%ile 


35 


36 


39 


42 


NCE 


42 


42 


44 


46 




SS 


528 


592 


638 


692 


Hartford 


%lle 


33 


35 


37 


48 




NCE 


41 


42 


43 


49 




SS 


580 


628 


673 


713 


PC Non-Match 


%lle 


57 


53 


54 


57 




NCE 


54 


52 


52 


54 



100 
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H Hartford -^'T ^^-^ (N=2d) 

PC Project Concern ^ ^ (N=26) 

PCNM Project Concern Non-Match (N=17) 



1 1 • 1 i 

Year Spring Spring Spring Spring Spring 

1981 1982 19S3 1984 1985 

Grade 3 4 5 6 7 

Figure 13. Spring 1981 to Spring 1985 MAT scores for suburban 

Project Concern and Hartford matched groups and the suburban 
Project Concern non-matched group: Reading. 



1U2 



60 



55 



50 



45 



NC£ 



40 



35 



PCNM r 



PC 
H 




30 



Year 
Grade 



f— 

Spr > ng 
1981 
3 



H Hartford {N = 26) 

PC Project Concern - ^ " (N=26) 

PCNM Project Concern Non-Match (N=17) 

*P<.05 



1 — 

Spri ng 
1982 
4 



Spri ng 
1983 
5 



Spri ng 
1984 
6 



Spri ng 

198 5 
7 



Figure 14 Spring 1981 to Spring 1985 MAT scores for suburban 
Project Concern and Hartford matched groups and the suburban 
Project Concern non-matched group: Math. 
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Table 31 

Spring 1981 to Spring 1985 MAT Scores for Project Concern 
and Hartford Matched Students and Project Concern Non-MaccHed Students 

Grade 4 (1980-1981 ) 



Group 



Spri ng 
1981 



Spr i ng 
1982 



Spr i ng 
1 983 



Spri ng 
1984 





ss 


671 


702 


737 


742 


Proiect Concern 




J o 


d 1 

H 1 


D u 


A A 


(N=18) 


NCE 


44 


45 


50 


47 




SS 


675 


709 


738 


734 


Hartford 


%lle 


40 


45 


50 


40 


(N=ie) 


NCE 


45 


47 


50 


45 




SS 


704 


726 


757 


750 


PC Non-Match 


%lle 


55 


55 


60 


48 


(N=21) 


NCE 


53 


53 


55 


49 




SS 


608 


637 


680 


729 


Project Concern 


%lle 


43 


37 


43 


47 




NCE 


46 


43 


46 


48 




SS 


604 


654 


711 


740 


Hartford 


%lle 


41 


45 


56 


53 




NCE 


45 


47 


53 


52 




SS 


626 


674 


708 


739 


PC Non-Match 


%ile 


52 


54 


55 


52 




NCE 


51 


52 


53 


51 



04 



lOo 
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Hartford 
Project Concern 
Project Concern 

* p*.05 



(N=18) 
(N=18) 

Non-Match (N=18) 



1 I I 1 I 

Year Spring Spring Spring Spring Spring 
1981 1982 1983 1984 1985 

Grade 4 5 6 7 8 

Figure 15. Spring 1981 to Spring 1985 MAT scores for 
suburban Project Concern ana Hcjrtford matched groups and 
the suburban Project Concern non-matched groups: Reading. 
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83 




Hartford (N-18) 

Project Concern (N=18) 

Project Concern Non-Match (N=21) 

* p<.05 



i I i i I- I 

Spri ng 
1985 
8 

Figure 16. Spring 1981 to Spring 1985 MAT scores for suburban 
Project Concern and Hartford matched groups and the suburban 
Project Concern non-matched group: Math. 
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Table 32 



Spring 1981 to Spring 1985 MAT Scores for Project Concern 
and Hartford Matched Students and Project Concern Non-Matched Students 

Grade 5 (1980-1981 ) 



Spring Spring Spring Spring Spr- 

Group 1 981 1 982 1 983 1 984 1 9( 







SS 


698 


730 


741 


759 


79- 




Project Concern 


%lle 


39 


46 


43 


41 


5' 




(N=27) 


NCE 


44 


48 


46 


45 


5' 






SS 


697 


723 


736 


761 


79; 




Hartford 


%ile 


38 


43 


41 


42 


5( 




(N=27) 


NCE 


44 


46 


:5 


46 


5( 






SS 


706 


751 


764 


776 


8o: 




PC Non-Match 


%ile 


43 


57 


55 


50 


5: 




(N=18) 


NCE 


46 


54 


53 


50 


5^ 






SS 


662 


707 


738 


777 


8o: 




Project Concern 


%lle 


49 


54 


52 


57 


6: 




NCE 


49 


52 


51 


54 


5: 






SS 


658 


712 


745 


787 


81< 




Hartford 


%lle 


47 


56 


55 


61 


7( 






NCE 


48 


53 


53 


56 


6: 






SS 


678 


718 


748 


769 


so; 




PC Non-Match 


%lle 


56 


59 


57 


53 


6^ 






NCE 


53 


55 


54 


52 


5£ 
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60 



55 " 



50 



45 



NCE 




40 



35 



30 



Hartford (N=27) 
Project Concern (N=27) 
Project Concern Non-Match (N=18) 



Spring Spring Spring Spring Spring 

1981 1982 1983 1984 1985 

5 6 7 8 9 

Figure 17. Spring 1981 to Spring 1985 MAT scores for suburban 
Project Concern and Hartford matched groups and the suburban 
Project Concern non-matched group: Reading. 
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Hartford 
Project Concern 
Project Concern 



(N=27) 
(N=27) 

Non-Match {f^ = 18) 



I I » • — 

Spring Spring Spring Spring 
1382 1983 1984 198 5 

0 7 8 9 

Figure 18. Spring 198"! Spring 1 985 MAT scores for suburban 
Project Concern and Hartford matched groups and the suburban 
Project Concern non-matched group: Math. 



Year ^P^^"^ 
1981 

Grade 5 



Overall 



After matching students on the basis of sex, ethnic 
group, Hartford sending school, and Spring 1981 
Reading and Math achievement levels, no significant 
differences were found between the Suburban Project 
Concern and Hartford comparison students from Spring 
1982 to Spring 1984 in Reading and Math and through 
Spring 1985 in Reading. Hartford 1981 grade 4 
students achieved significantly (p<.05) higher levels 
than Suburban Project Concern students in 1985 grade 
8 Math scores. 

The findings for the 1981 grade levels were as follows: 

1981 Grades 3 -5 

The grade 3 1981 matched Suburban Project Concern and 
Hartford groups remained below grade level (i.e., 
below a NCE of 50 or 50th %ile)'in Reading and Math 
from grade 4 (1982) to grade 7 (1985). The highest 
performance level for both groups was exhibited 
during grade 6 (1984) with a decline evident during 
grade 7 (1985) (Table 30, Figures 13-14). Both 
groups sustained their 1981 achievement levels in 
Reading and Math. 

The grade 4 1981 matched Suburban Project Concern and 
Hartford comparison groups were below grade level in 
grades 5 (1982), 7 (1984) and 8 (1985) and at grade 
level in grade 6 (1983) in Reading. The Hartford 
comparison students exhibited above grade level 
performance in Math beginning Spring 1983 (end of 
grade 6) and exhibited significantly higher Math 
c^chievement (p<.05) than the Suburban Project Concern 
students during grade 8 (1985) (Table 31, Figures 
15-16). Both groups sustained their 1981 achievement 
levels in Reading and Math. 

The grade 5 1981 matched Suburban Project Concern and 
Hartford comparison groups both remained below grade 
level from grade 6 (1982) to grade 8 (1984) and 
exhibited grade level performance in Reading in grade 
9 (1985). In Mathematics both groups exhibited above 
grade level performance from grade 6 (1982) to grade 
9 (1985); continual improvement was found for both 
groups for grade 8 (1984) and grade 9 (1985) (Table 32, 
Figures 17-18). Both groups sustained their 1981 
cxhievement levels in Reading and Math. 
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In summary, this section has presented empirical evidence which indicates 
that, overall, the matched Suburban Project Concern and Hartford comparison 
groups tended to sus .in their 1981 achievement levels from 1982 to 1985. 
With one exception there were no differences between Suburban Project Concern 
and Hartford students in Spring 1982 to Spring 1985 Reading and Math 
achievement levels after matching the groups on sex, ethnic group, Hartford 
sending school, and Spring 1981 achievement levels. In the one case where 
differences existed, Hartford students axhibited significantly higher levels 
of Math achievement than Suburban Project Concern students. These compari- 
sons represent the primary analyses for the reseL.'ch question stated earlier 
in this chapter We will now turn to the secondary issue of examining the 
achievement levels of the Subu'^ban Project Concern students who had no 
matches in the Hartford sending schools - the Project Concern Non-Match group. 

Project Concei-n Non-Match versus the Suburban Project Concern and 
Hartford Matched Groups . Tables 30-32 present the achievement data for the 
Project Concern Non-Match group. These data are depicted in Figures .13-18. 
While t.,a Snring 1981 Reading and Math achievement levels of the Non-Match 
group visually appear higher than the Project Concern and Hartford Matched 
groups, with two exceptions, no significant differences were found among the 
three groups for the Spring 1981 data. The two exceptions were as follows: 
the Non-Match group axhibited significantly higher Spring 1981 achievement 
than the Hartford group for grade 3 Math (see Figure 13) and significantly 
higher Spring 1981 achievement than the Project Concern group for grade 4 
Reading (see Figure 15). For the Spring 1982 to Spring 1985 data readers 
should note that no significant differences wv»re found between the Project 
Concern Non-Match and either the Suburban Project Concern or the Hartford 
groups. Overall, the Project Concern Non-Match group sustained its 1981 
achievement level from 1982 through 1985. 

1L3- 
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In summary, these findings are Important for our consideration. They 
indicate that the group of Suburban Project Concern students who had no 
matches in their Hartford sending school (based upon sex, ethnic group and 
achievement) were above grade level in Spring 1981 Reading and Math 
achievement, tended to remain in most cases above grade level from Spring 
1982 to Spring 1985, but did not significantly outperform the Project 
Concern or Hartford Matched students during the 1982 to 1985 years* 

Overall Sustained Achievement Effects , In the previous sections we 
have presented data which indicates that, overall, the 1981 achievement 
levels were sustained for the Suburban Project Concern, Hartford Non- 
Participants and Project Concern Non-Match groups. We noteo that the primary 
objective of Study III was to examine the differences in achievement levels 
between and among the respective groups for the 1981 to 1985 test times. 
Now that we have accomplished this objective we will offer some exploratory 
observations regarding the achievement trends depictnd in Figures 13-18, 
Profiles of student achievement levels such as those presented in Figures 
13-18 can appear to depict a rise or fall in achievement from year to year. 
Explaining these changes in the slope of the figure is often difficult and 
sometimes impossible. The change could reflect a subtle change in a test 
form or, perhaps more often, changes in curricular or instructional emphasis* 
When only one school system is involved, it is often possible to analyze the 
match between the curriculum and the test content. As we noted earlier in 
Study I, the fact that several school districts are included in these datd 
makes the situation harder to explain. 

Fro purely exploratory approach we examined Figures 13-18 for any 
obvious V al changes in achievement levels from 1981 to 1985. School 
personnel may -/ish to discuss these observations further to ascertain if any 
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meaningful explanations are possible. Table 33 presents a suiimary of the 
most noticeable changes in the figures using the "squint your eyes and look 
at the figure" approach. For the 1981 grade 3 group (Figures 13-14) 
increases during grade 6 and decreases during grade 7 were observed in 
Reading and Math. For the 1981 grade 4 group (Figures 15-16) increases were 
observed during grade 6 in Reading and Math; decreases were observed during 
grade 7 in Reading. Finally, for the 1981 grade 5 group increases were 
observed during grade 9 in Reading and Math, Again, we note that these 
changes in the slope of the achievement level over time may not be meaningful, 
but could be worthy of some discussion by staff from curriculum and instruc- 
tion perspectives. 
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TabU 33 
Summary of Possible Changes In 
Achievement Trends by 1980-1981 Grade Level 



198Q-1981 

^''^^^ With Increase 


Area 


Grade 
VJith Decrease 


Area 


3 6 


Readi ng.Math 


7 


Readi ng ,Math 


4 6 


Readi ng ,Math 


7 


Readi ng 


5 9 


Reading, Math 
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CHAPTER IV 
SUMflARY 

The evaluation of the 1984-85 Project Concern program focused on the 
following two areas: 

1. Monitoring the cognitive and affective impact of Project Concern 
over the current school year, 

2. Examining the extent to which the reading and mathematics achieve- 
ment gains of various groups of Project Concern students at grades 
3, 4, and 5 were sustained over time. 

The purpose of this summary is to collate for the reader some of the 

major findings of this evaluation. It is important to note that perceptions 

of the Project Concern program should not be formed on the basis of this 

summary alone. All findings must be interpreted m light of the evaluation 

design utilized, a more complete discussion of the r^^ults presented, and the 

limitations placed on the findings obtained. 

Monitoring the Cognitive and 
Affective Impact of Project Concern 

The impact of Project Concern on the cognitive achievement of program 
participants was assessed by comparing the Metropolitan Achievement Test 
results from spring 1984 to those obtained during spring 1985. Achievement 
growth on an absolute basis addressed the question - how much basic skill 
growth have Project Concern students exhibited? A summary by grade level of 
skill areas where Suburban and Inner-City Project Concern participants 
exhibited statistically significant absolute growth is presented below. 
While statistically significant absolute growth was exhibited by Subu^^ban 
Project Concern students at most grade levels for the three skill areas, 
this was not the case for Inner-City participants. The lack of statistically 
signficant achievement growth for Inner-City participants, especially at grades 
4 and 7 can be explained by the very small sample sizes at these grade levels. 
(6r. 4, N=7; Gr. 7, N=9). With such small sample sizes, it is difficult for 
a group of students to exhibit a statistically significant achievement growth. 
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Suburban Component^ 




Grade 


Keacn ng 


Language 


i Id L 1 icitia L 1 ^ o 


3 








4 








5 




** 




u 








7 








8 








9 






ieie 


10 










Inner-City Component 






Reading 


Language 


Mathematics 


4 








5 


* 






6 








7 








8 


* 







In reviewing the results of the 1983-84 Evaluation of Project Concern, 
Superintendents in participating ccnmunities noted that it would be helpful 
to provide information regarding the absolute growth of Project Concern 
students using grade equivalent scores. This information is provided on Page 94 
for the Suburban and Inner-City Components of the Project Concern Program. 

In reviewing these results, it is important to note that the grade level 
indicated is for Spring 1985. For example, at grade 3, 28 Suburban Project 
Concern students were tested in the Spring 1984 when they were in grade two, 
and then they were tested again Spring 1985. Their mean or average pretest 
language performance in Spring 1984 was 2.8 and their average post test perfor- 
mance in Spring 1984 was 3.5. Thus they exhibited a growth in language of 7 
months between pre- and post testing. 
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^Note: * indicates absolute growth is statistically significant at the 
.05 level. 

** indicates absolute growth is statistically significant at the 
.01 level. 
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Suburban Component 









Reading 






Lianguage 




Mathematics 


Grade 


N 


Pre- 


Post 


Gain 


Pre- 


Post 


Gain 


Pre- 


Post 


Gain 










1 


2 8 


3 5 


7 


2.9 


3.2 


.3 


A 
■f 




*i 1 

0*1 




Q 


3 5 


5 0 


1 5 


3.6 


4.3 


.7 


D 


HP 


J • o 








5 8 


1 


4.8 


5.6 


.8 


6 


49 


5.6 


7.6 


1.0 


6.0 


7.8 


1.8 


5.6 


7.0 


1.4 


7 


66 


6.4 


/ • J 


. y 


/ . i 


o .U 


Q 


6.7 


7.3 


.6 


8 


68 


7.3 


9a 


1.8 


8.4 


9.5 


1.1 


7.7 


9.0 


1.3 


9 


52 


8.6 


10.3 


1.7 


10.2 


11.8 


1.6 


9.7 


12.3 


2.6 


10 


60 


9.7 


10.1 


.4 


10.3 


10.5 


.2 


10.8 


10.8 





Inner-City Component 



Grade 


N 




Reading 






Language 




Mathematics 


Pre- 


Post 


Gain 


Pre- 


Post 


Gain 


Pre- 


Post 


Gain 


4 


7 


3.3 


3.5 


.2 


5.2 


5.9 


.7 


4.3 


4.4 


.1 


5 


17 


4.5 


5.6 


1.1 


5.4 


5.4 


0 


4.8 


6.2 


1.4 


6 


16 


5.3 


7.0 


1.7 


6.4 


7.5 


1.1 


6.3 


7.7 


1.4 


7 


9 


5.8 


5.4 


-.4 


6.7 


6.6 


-.1 


6.8 


7.0 


.2 


8 


15 


7.1 


8.6 


1.5 


7.1 


8.3 


1.2 


7.4 


9.2 


1.8 
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When reviewing the grade eqt^ivalent score infonnation provided, it is 
Important to keep in mind t^e definition of a grade equivalent score. A 
grade equivalent score of ^..8 in reading for Suburban participants indicates 
that the performance of Project Concern participants was the same as the 
average performance of students nationally at the second grade in the eighth 
month of school who participated in the standardization of the Metropolitan 
Achievement Tests. One cannot conclude that students have mastered all skills 
necessary to read graded text at the 2,8 level- This pofnt is particularly 
important to keep in mind when reviewing the math growth of Suburban Project 
Concern participants at grade 9. The average performance of these students 
when they were in the eighth grade during Spring 1984 was 9 J. In Spring 
1985 the average language performance of these same students was 12,3, 
indicating an average growth of 2 years and 6 months. The Spring 1985 score 
of 12.3 does not indicate that these students have mastered math skills up 
to the 12th grade, but rather that they received the same score on this test 
as the average 12th grader. In fact, the level of the Metropolitan Achieve- 
ment Tests used with these students was not developed to assess math skills 
at the 12th grade. 

From the information provided, it is evident that both Suburban and 
Inner-City Project Concern students exhibited considerable basic skill 
progress as indicated by their grade equivalent score growth at most grade 
levels in reading, language, and mathematics. The amount of progress varied 
by skill area and by grade level. When reviewing the grade equivalent score 
results, it is beneficial to examine progress at a particular grade level 
across the three skill areas of reading, language, and mathematics. This 
approach provides a more informative assessment of Project Concern partici- 
pants' basic skill growth- Suburban and Inner-City Project Concern partici- 
pants exhibited appreciable grade-equivalent score pains at most grade levels 
where statistically significant achievement growth was exhibited. 

120 



In addition to examining the absolute basic skill achievement growth 
of Project Concern participants, relative achievement growth was also 
examined. 

In assessing relative growth, one is asking the question - as a result 

of the achievement progress exhibited in the areas tested, has the relative 

standing of the students changed regarding the national norm group? The 

following findings resulted from this assessment: 

Inner-City Project Concern participants tended to exhibit 
relative basic skill growth in those areas at each grade 
level where statistically significant absolute growth was 
evident. This indicates that the statistically signifi- 
cant basic skill progress exhibited by these students was 
generally reflected in an imporvement in their standing 
relative to the national norm group. 

Inner^City Project Concern participants exhibited an overall 
relative NCE gain In Mathematics (+0.5) and losses in Read- 
ing (-0.7) and Language (-3.9). 

Inner-City Project Concern participants maintained or 
exhibited NCE gains in all three basic skill areas at 
grt:des 6 and 8, while students at grades 4 and > 
exhibited NCE losses in these areas. Pre- post NCE 
achievement test results for students at grades 5 were 
mixed. 

Suburban Project Concern participants tended to exhibit 
relative basic skill growth in reading at each grade 
level where statistically significant absolute growth was 
evident. For reading, the statistically significant 
basic skill growth exhibited by students was usually 
reflected in an improvement in their standing relative to 
the national norm group. This was not the case for the 
skill areas of mathematics and language. 

Suburban Project Concern participants exhibited an overall 
relative NCE gain in Reading (+0.6) and losses In Language 
(.0-5) and Mathematics (-K2). 

Suburban Project Concern participants exhibited NCE gains 
in all those basic skill areas at grades 6 and 9, while 
students at grades 3 and 10 exhibited NCE losses In those 
- three areas. Pre post NCE achievement test results for 
students at grades 4, 5, 7, and 8 were mixed. 



EKLC 



121 



Further insights into the relative basic skill s>'Owth of participants 
was obtained by grouping students on the basis of their spring 1984 percen- 
tile ranks. The following trends were found: 

For Inner-City Project Concern participants, students at 
or below the 36th percentile at most grade levels tended 
to exhibit the most NCE growth in Reading and Mathematics. 

For Suburban Project Concern participants, students at 
or below the 50th percentile at most grade levels tended 
to exhibit the most NCE growth in Reading and Mathe- ^ 
matics. 

Another way of looking at the relative growth of Project Concern 
participants is to use overall percentile ranks. By calculating the mean 
or average pre- and post Metropolitan Achievement Test scores of all 
Project Concern Participants for the skill areas of reading, language, and 
mathematics, insights can be gained regarding the question - How did Project 
Concern students as a group progress in their basic skill development? This 
percentile score information is provided below. 

Percentile Ranks 



Skill Areas Suburban Inner-City 





Pre 


Post 


Pre 


Post 


Reading 


47 


48 


45 


44 


Language 


54 


. 53 


55 


48 


Math 


51 


49 


52 


53 



From the infonnation presented, it is evident that Suburban Project 
Concern students scored at the 47th percentile on the pretest In Reading 
during the Spring 1984. By scoring at the 47th percentile, this means that 
the average Suburban Project Concern Program participant's performance In 
reading was better than or equal to 472 of the students in the national 
norming sample for the Metropolitan Achievement Tests. It is also useful to 
note that the 50th percentile is equivalent to grade level performance. 
Keeping these points in mind, it is evident that on the average, both Suburban 
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and Inner-City Project Concern participants scored below grade level on the 

pre- and post test in reading, but they were at or above grade level on the 

pretest in mathematics and language. On the post test, Suburban participants 

remained above grade level in language, but drop below grade level in Math. 

Inner-City participants cropped below grade level on the post test in 

language, but remained above grade level in Math. .Suburban Project Concern 

participants exhibited a slight decrease in percentile rank from pre- to post 

testing in all skill areas except reading. Inner-City exhibit a percentile 

decrease in all skill areas except for math. 

To examine che affective area, the Student Survey was administered 

during the spring of 1985 to participants in the Suburban and Inner-City 

components of Project Concern at grades 2-10. With respect to differences 

in self-concept and school attitudes across grade levels, some significant 

differences similar to previous years* data for Suburban participants were 

evident as follows: 

As grade level increased, more students tended to feel 
that school work was fairly easy for them (item 1). 

As grade level increased, more students felt they could 
get good grades if they wanted to (item 3). 

As grade level increased, fewer students indicated they 
often volunteer to do things in class (item 4). 

As grade level increased, fewer students were proud of 
their school work (item 7). 

As grade level increased, more students felt ;:ney were 
not doing as well in school as they would like to do. 

For Inner-City participants, the responses differed significantly across 

grade levels for only item 1. As grade level increased, fewer students 

tended to feel that school work was fairly easy for them. 
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In sumnary, it can be concluded that the self-concept and school 
attitudes of the Suburban and Inner-City Project Concern students in the 
areas of school and school work, classroom participation, and teachers are 
quite positive. The affective orientation of students participating in the 
1984-1985 Project Concern Program is fairly consistent with the results of 
past evaluations of Project Concern when the Student Survey was used. 

Examining the Sustained Cognitive 
Effects of Project Concern Participation 

These sustained cognitive effects studies were conducted as part of the 
1984-85 Hartford Project Concern Program Evaluation. The evaluation 
question addressed in each study as well as a summary of the major findings 
are presented in the subsequent sections of this chapter. 

Study I 

How are students performing who were in grades 3-5 
during the 1980-1981 year and continued in the program 
through the 1984-1985 year (graces 7-9)? 

In conducting this sustained cognitive effects study, the spring 1982 to 
spring 1983. spring 1983 to spring 1984, and spring 1984 to spring 1885 
sustained Reading and Mathematics achievement gains for Project Concern 
participants were examined in relation to gains made from spring 1981 to 
spring 1982. This study focused on students who were enrolled at grades 3-5 
during the spring of 1981 • On the basis of norm group comparisons using NCE 
scores, the following findings were forwarded for the Suburban and Inner-City 
students. 

Total Group 

Reading . Grades 3, 4 and 5 (1981) increased their 
1981-1982 gains in Reading performance. 
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Hath . Grade 3 (1981) exhibited an overall decline in 
HatF oerformance as of grade 7 (1985); grade 4 (1981) 
and grade 5 (1981) exhibited overall increases in their 
1981-1982 Math performance gains through grade 8 (1S85) 
and grade 9 (1985) respectively. 

Suburban 

Reading . Grades 3, 4 and 5 (1981) increased their 
198i-1982 gains in Reading performance. 

Math. Grade 3 (1981) exhibited an overall decline U 
MatF performance as of grade 7 (1985); grade 4 (1981) 
and grade 5 (1981) exhibited overall increases in their 
1981-1982 Math performance gains through grade 8 (1985) 
and grade 9 (1985) respectively. 

Inner-City 

Reading and Math . Grade 3 (1981) students exhibited 
an overall decline in Reading and Math performance as 
of grade 7 (1985); grade 4 (1981) students exhibited 
overall increases in their 1981-1982 Reading and Math 
performance gains as of grade 8 (1985). 

Overall, the findings of this study support the claim that Project 
Concern students have generally maintained and enhanced their achievement 
performance over time. The exception to this general trend was found for 
all three cohort groups during grade 7. 

Study II 

How are students performing who were in grades 3-5 
during the 1982-1983 year and continued in the program 
through the 1984-1985 year (grades (5-7)? 

In conducting this sustained cognitive effects study, the spring 1984 
to spring 1985 sustained Reading and Mathematics achievement gains for 
Project Concern participants were examined in relation to gains made from 
spring 1983 to spring 1984. This study focused on students who were enrolled 
at grades 3-5 during the spring of 1983. On the basis of norm group 
comparisons using NCE scores, the following findings were forvarded for the 
Suburban and Inner-City students. 
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Reading and Math . Grade 4 students increased their 
1983-1984 gain during the 1984-1985 year; grades 3 
and 5 students exhibited decreases in their 1984- 
1985 achievement levels when compared to their 

1983- 1984 levels. 

Suburban 

Reading . Grades 3 and 4 students increased their 1984- 
1985 achievement levels after exh-<biting a decrease 
during the 1983-1984 period. Grade 5 students failed 
to maintain their 1983-1984 gains during the 1984-1985 
year. 

Math. Grade 4 students increased their 1983-1984 gains 
during the 1984-1985 year. Grades 3 and 5 students 
failed to maintain their 1983-1984 gains during the 

1984- 1985 year. 

Inner-Cit y 

Readi ng. Grade 4 students increased their 1983-1984 
gains during the 1984-1985 year; grades 3 and 5 
students failed to maintain their 1983-1984 gains 
during the 1984-1985 year. 

Mat]i. Grades 3 and 4 students exhibited gains during 
1^4-1985 as compared to their 1983-1984 achievement 
levels; grade 5 students failed to maintain their 
1983-1984 gains during the 1984-1985 year. 

The results of this sustained effects stUy are based on a one year 

(Spring 1984 to Spring 1985) period. When compared to the four year period 

of data presented in Study I, these sustained effects results are considered 

to be for a relatively short period of time. It was found that over the one 

year (Spring 1984 to Spring T985) period several of the Spring 1983 to 

Spring 1984 gains were not maintained. It will be necessary to follow these 

same students over a longer time period before any meaningful trends In 

achievement can be noted with confidence. 
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Study III 

How does the achievement of Suburban Project Concern 
students, who were in grades 3-5 during the 1980-1981 
year and continued in the program through the 1984-1985 
year (grades 7-9), compare with a comparable group of 
Hartford students not participating in the program? 

The evaluation design employed to answer tnis question is similar to . 
the traditional sustained achievement effects design employed In Study I. 
The unique futures of Study III are the use of a Hartford comparison group 
and the emphasis on the statistical significance of the differences in the 
achievement levels of Project Concern and Hartford comparison students at 
each of the 1981 to 1985 test times. The overall and grade level findings 
of this study were as follows: 

Overall 

After matching students on the basis of sex, ethnic 
group, Hartford sending school, and Spring 1981 
Reading and Math achievement levels, no significant 
differences were found between the Suburban Project 
Concern and Hartford comparison students from Spring 
1982 to Spring 1984 in Reading and Math and through 
Spring 1985 In Reading. Hartford 1981 grade 4 
students achieved significantly (p<.05) higher levels 
than Suburban Project Concern students in 1985 grade 
8 Math scores. 

1981 Grades 3-5 

The grade 3 1981 matched Suburban Project Concern and 
Hartford groups remained below grade level (i.e., 
below a NCE of 50 or 50th %ile) in Reading and Math 
from grade 4 (1982) to grade 7 (1995). The highest 
performance level for both groups was exhibited 
during grade 6 (1984) with a decline evident during 
grade 7 (1985). Both groups sustained their 1981 
achievement levels in Reading and Math. 

The grade 4 1981 matched Suburban Project Concern and 
Hartford comparison groups were below grade level In 
qrades 5 (1982). 7 (1984) and 8 (1985) and at grade 
level in grade 6 (1983) in Reading. The Hartford 
comparison students exhibited above grade level 
performance in Math beginning Spring 1983 (end of 
grade 6) and exhibited significantly higher Math 
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achievement (p<.05) than the Suburban Project Concern 
students during grade 8 (1985). Both groups sustained 
their 1981 achievement levels in Reading ard Matn. 

The qradA 5 1981 matched Suburban Project Concern and 
HartVord comparison groups both remained below grade 
level from grade o (1982) to grade 8 (1984) and 
exhibited grade level performance in Reading in grade 
9 (1985). In Mathematics both gi-o^PS e^jl^^i*** 
gride level perfomance from gr::de 6 (1982) to grade 
9 (1985); continual Improvement was Jound for both 
groups for grade 8 (1984) and grade 9 (1985). Both 
groups sustained their 1981 achievement levels In 
Reading and Math. 

In surnn^ry. these findings indicate that, overall, the matched Suburban 
Project Concern and Hartford comparison groups tended to sustain their 1981 
achievement levels from 1982 to 1985. With one exception there wer3 no 
differences between Suburban Project Concern and Hartford students in Spring 
1932 to Spring 1985 Reading and Math achievement levels after matching the 
groups on sex. ethnic group. Hartford sending school, and Spring 1981 achieve- 
ment levels. In the one case at grade 8 where differences existed. Hartford 
students exhibited significantly higher levels of Math achievement than 
Suburban Project Concern students. 
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